Anybody know what the formula or dataset field I use to get the subtotals of transactions?

Formula (Numeric) = nvl({totalamount},0)-nvl({discountamount},0)-nvl({taxtotal},0)-nvl({shippingamount},0) Label = Subtotal

But does that work in Suiteanalytics datasets? i know that works in saved searches.

Yes, I believe it works. Just verify if the IDs are the same as those on the datasets, which can sometimes differ. But it's a fairly simple formula.