Is it possible to make available the standard email functionality under communication subtab that is available in transactions such as sales orders to a given custom transaction? Or the only way in this case is to use the N/email module?

Mail Merge is enable in the account. What I have seen is that you can enable mail merge in custom record types. However, I do not see that option for Custom Transaction Type, in the settings the custom transaction itself. You do have the communications subtab, but there are no email options as in other transactions. Also, I do not see anything related to mail merge in instance of the custom transaction. Is this what you meant?

whoops, i thought this was for a custom record

So far I think is not possible, so I might have to resort to some custom stuff for the email functionality. Thanks!