We created a custom PDF Layout and set it as preferred, the transaction form says the printing type is basic, why is it not using the custom layout?

It sounds like there may be an issue with the transaction form not recognizing the custom layout as the preferred option. I recommend checking the setup of the transaction form to ensure that it is properly configured to use the custom layout.

Figured it out, there was a subsidiary setting overriding the form setting (for printing checks)