Is there a way to customize a role in a way where the role is restricted to viewing only one product group/class, or one vendor's items, or one customer's transactions?

I don’t believe you can do so natively. Though, you could do this with scripting.

Thank you - a simple script or more complex?

Nothing too crazy— Essentially a beforeLoad script that checks the customer ID of the record being loaded. If the customer ID doesnt match the single customer ID that you want the user to be able to see then it could throw an error “You dont have permission to view this customer” Then the same idea for the other two record types

Thank you!

Keep in mind this won't prevent them from seeing such record in searches/lists