I've been struggling with that for years. It seems that custom sublists (from the 'Record is Parent' checkbox on a custom field setup) can be re-ordered on Custom Entry Forms but not Custom Transaction Forms; only standard sublists appear in the Custom Form setup on the latter. I seem to recall an Enhancement Request related to that but now I can't track it down (not that voting on those seems to ever actually make a difference...).