If you return the raw paged results set getInputData may be able to handle that without any problem.

Raw paged results is larger than 500mb and takes 8 hours before NS times out

can you use

runSuiteQLPaged

on 'SELECT id FROM customrecord___' to get all the custom record internal ids? like inside of getInputData, and then pass id array to map idk if that would still timeout or be too large of data or can include additional WHERE clause to split out in separate more manageable batches, if doesn't need to be done all at once for the 4mil

Yeah this would be just as productive as doing it as a search or dataset filter

been a while since messed with it, but I think there's also stuff can do with

N/task

to export entirety of saved search to like CSV of ids in fileCabinet or something as well - just to get the basis of ids to iterate over - multiple routes

Oooh that’s a great suggestion

@dbarnett is correct. Running a saved search task asynchronously allows you perform searches with massive result sizes, and it's quite fast. Probably a better way to do this, but you can also make it run synchronously by checking the status iteratively. @chris

if I run a paged resultset, the result that's returned to the Map stage is done concurrently or serialized?

Why not start the MR with the first 100 pages or so of the search and let it run? Then in summarize, schedule itself if there are more results in the same search. Sure, it will take many reschedules so there will be some overhead for that. But the time for deletion will be limited to how many processors you have access to. Even if you can output the results to a csv with N/task, considering the memory limit of the mr, would it even be able to load it?

I have over 30~ processors to safely reserve, and I think hard limiting pagination makes sense, too