Is there any way to cancel sales order by script or workflow?

You can use a WF to mark the closed box on every line of the SO

You may be able to set the status via SuiteScript. I know you can for other statuses. For example, we have a WF Custom Action that (as step one of MANY things it does) sets the status to approved for a sales order as follows:

bo.setValue({fieldId: 'orderstatus', value: 'B'});