Hi, so that's more of a correct approach question rather than specific code. I have M/R script which processes number of records. During the execution I need to add/modify some data into a global object/storage, then do some additional record processing in the summary stage. I thought about using runtime.Session.set(), but that gets reset on each stage. I also thought about using custom record to store JSON data, but there's a limit of 4000 characters for textarea field. Finally cache, but it's not really guaranteed to persist. The only solution I can think of would be temporarily storing data in a file, but I refuse to believe there's no better way. Any ideas?

the simplest approach is to switch to a Scheduled script instead and hold the data in memory.

But then 10000 usage units may not be enough.

Can you not stop at 10,000 and pick up where you left off?

If the data can be split up into rows, can you store a number of custom record rows, then have a 2nd M/R that deals with them?

Well, I went the longer way around... Basically I'm passing total number of records from getInputData(). Session doesn't get wiped within single stage, so I'm building data object in reduce stage (this works fine this time as each map() key is unique). Then in reduce stage, if current key is equal to total keys available, I'm passing additional data from session to context and effectively to summary stage. Unfortunately it looks like you can't overwrite context under the same key within single stage, therefore I need to wait for last element to be processed.

I'm curious why you need to globally store something that is changing with steps in either map/reduce. Why don't you pass whatever that information is down through the stages and consolidate the data in the summary step?

It's not just storing, it's storing and updating data object along with each step, based on results of operations in current stage

If you need to read from this object during the map/reduce stages that sort of implies that the order they are processed in somehow matters. As Shawn already stated, sounds likely a Scheduled script is the more appropriate approach. If you do not need to read from this data object iteratively and only need to write/update it, then you should be able to just do that in summary instead.

Do the stages actually update anything themselves along the way, or are you just wanting to do 1 update/creation at the end?

No, so maybe to make things bit more clearer - I'm going through number of custom records, gather information about existing items and SOs , then at the end, based on gathered information, decide which/what type of Invoices need to be created or updated and just do it in one go. For now I just wanted to construct data object containing the data and do record transform/creation/update in summarize(). Updating Invoice on each run of particular stage would be a nightmare. Maybe Scheduled script is the way to go...

Sounds like you can do the first bit in a SuiteQL query

I got the getInputData() covered already, it's about going through results in map/reduce and gathering all information into one place

longtext limit is 100,000, you may wanna try that

Anyway, thanks guys for your support, have a good weekend 🙂

you could try defining a const obj, and each map stage look for the const obj, and if it's blank run the job that looks for the const obj. the const obj from what i was told will last until the script yields. If your lookup job isn't massive throughout one session the const job will last so you can reduce runtime by only looking up on once of the map jobs instead of repeating for every map job. problem is everytime it yields it will reset, meaning you need to setup a mechanism to refetch if it empties