can we use a custom field (on a transaction) to join a saved search to an entity as a custom sublist?

I'm going to say "yes". Custom Field will need to be of a type List/Record.

is there a maximum number of columns allowed in the saved search for sublist to show?

let us know if you find out

Seems to be no limit. I got my sublist to show eventually but not related to number of columns.