I do not know if it is possible in NetSuite settings to control rounding on amount column but maybe the tax calculation should be on the amount before it gets rounded up in the SO.