you might try throwing a TO_DATE function on {customitemdatefield} to to make sure.
You might also consider something like this CASE WHEN TO_NUMBER(TO_CHAR(TO_DATE({customitemdatefield}),'YYYYMM')) between 201707 and 201806 THEN {amount} ELSE 0 END.
In my experience, working with dates in Julian format is much better when doing math. Also, you don't need YYYYMMDD since it appears you are looking for something less granular.