Has anyone ever connected to a SFTP server by using public/private key? I'm getting an error with the 'hostkey' parameter

You have to generate the key in an encryption format NetSuite expects. There are some help docs to walk through it

Command Promt: ssh-keygen -t rsa - PEM