Not sure about custom fax messages. AFAIK there's no API for sending a generic fax; however, if you want to fax say an Invoice, you should be able to simply check the

tobefaxed

box on the Transaction record. From Records Browser:

Check this box if you want to fax this transaction. Enter the fax number in the space to the right of the check box if it doesn't already appear. To fax NetSuite forms, an administrator must first set up fax service at Printing, Fax and Email Preferences.