I have a custom sublist under the related records tab, i would like it to show up first and opened before related records and transformations, is there any way to do that?

Try to edit your custom form > sublists > related records I think you should be able to rearrange the sublists by dragging them around:

my custom sublist does not show up in that the tab

@reuben, still having trouble with this?

Yes I still have not figured out how to switch the order through the netsuite ui