You can't restrict, however, you can create a custom field that has the users you are want to be available in the list. Populate that field with the user on creation, and csv import to update old ones. Then use this field as the filter since it has a custom list of employees. Kinda convoluted....but definitely doable if it's required