I created a custom transaction column field with List/Record type sourced from Item list. I created a saved search for custom sublist, so that all transactions associated with the item selected in that CUSTOM field will appear in the ACTUAL item record. It's not working. Is it because the custom field is not the native item field? Or I'm just doing something wrong because it's totally possible?

Filtering of your Custom Field values is done on the setup of the Custom Field, Tab Sourcing and Filtering. Add your search criteria there.