Is there a way to have a non grouped search, but still do a criteria based on a group? I need individual invoices in the list, but I only want to return invoices if the customer has 3 or less open invoices

You could add a field to the Customer record called "Number of Open Invoices" then have that field populated by a saved search/workflow. Then your saved search can show all invoices for customers where the Number of Open Invoices <=3