Oh, I think I see a possible problem. Since you have MAX() around {item.quantityonhand}, that will force you to put all other values inside of MAX(), MIN(), SUM(), COUNT(), or some other grouping function. You can't use MAX on just one part of a SQL select statement. You shouldn't need MAX at all.
Does this work:
CASE WHEN ({item.quantityonhand} - (CASE WHEN (TRUNC({today}) - {trandate}) < 30 THEN {quantity} ELSE 0 END)) < 1000 THEN 777 ELSE 888 END