Is it not possible to use the fields of a custom record on an invoice pdf?

depends on how the custom record is linked with the invoice. If just custom field with list of custom record (1:1) then you should be able to print that. If parent child relationship, you need some other way to get the data as there can be 1:M. You can have a saved search to show custom record values on invoice (long text field on invoice) or other scripted solutions.

It's one to one. I've tried ${record.fieldIdOfCustomRecordOnInvoice.fieldIdOfFieldOnCustomRecord} but it isn't working

Then create a custom field that sources that value (store value = false) from custom record and use this new field in the pdf

It's like 30 fields or so, so I'm trying to avoid it