Have a custom record (type X) that has a sublist, the elements of which are of custom record type Y. How to create an instance of X with the sublist populated with some instances of Y?

Create the instance of X, then create instances of Y that reference the instance of X in the field of Y designated as a parent record.

If Y is a child of X, you can just create X then reference Y as a sublist. Something like 'recmach<fieldid>'. Where <fieldid> is the field in Y referencing X. Using this approach also saves you governance units.