is there a way to reference multiple line items on a sales order as a criteria (ie a sales order must have product x, product y, and product z ---> then show up on the search) ??

I think using multiple instances of "Item On Any Line" would do the trick. One instance for each item you're looking for.

can you do a custom field with certain data 'on any line'?

Like a custom transaction line field?