I think the combo of Maximum and Sum is fine, but I'm thinking it doesn't like that you're only summing part of your formula. I think when you're doing that kind of summary combination, it's all or nothing in the formula. So you'd have to do either sum({custrecord161}-{custrecord154.shippingcost}) or sum({custrecord161})-sum({custrecord154.shippingcost}).
First option I'm fairly sure would work. 2nd one I'm not as confident in.
I'd also double-check that both 161 and 154 are the same type and that they're both numeric. I think it throws unexpected error on type mismatches sometime too.