i am unsure how MAX in formulas work in isolation within a saved search, because the behavior depends entirely upon how Netsuite generates the GROUP BY statement. With no group by, MAX(any_column) will return the largest value for that column across all records returned by the search criteria. That may, or may not be the value you want. I would definitely break the formula down into it's constituent parts, and display them separately, so you can work through the output values and math, to make sure everything is working as expected.
I would create 5 separate formulas for the following, to help troubleshoot:
{item.quantityonhand}
{quantity}
TRUNC({today})
TRUNC({today}) - {trandate}
(TRUNC({today}) - {trandate}) < 30