Is it possible to submit a suitelet form and have it start downloading a generated file instead of going anywhere? Or would it be better to add a "download" button and just use regular javascript to do it?

I've had luck using a button connected to a client script that generated a text file and immediately downloaded.

You can do it on suitelet as well just use writefile