Hi everyone, i have a scenario of a Map/Reduce that runs daily and update ~20k records but once in a while it would skip 10-15 records. I'm not sure how it would happen. What could we do to avoid that?

Honestly, it's plenty possible you're looking at a momentary or even a recurring problem on NetSuite's end, but as developers we have to pretend like that's not the case. 😂 I would just assume there's something wrong with the criteria it's using to collect records.

no error message on the 10-15? any time related info about those? were they created/edited WHILE the script was running?

Does it have the proper error handling?

I got no errors at all. Seems like the script just skipped them. If I ran the script again, they went through just fine

check system notes on the 10-15 that were skipped, compared timestamps with the runtime of the MR script and see if any of the system notes show a change in that time frame where a data element was modified that's part of your search filters

There's no system notes on the 15 records skipped either

Time zones? I would check the schedule / repeat every

more information might help what record type? is it search? or query? something else? what are the filters/WHERE clause?

Logging Errors at the summarize stage is what you need to do to rule out errors

Thanks @battk, that's helpful. Another question, i'm just curious if setting concurrency of 8 or more contributing to this issue?

i have neither the code nor the error message, you get the boring it could be

generally increasing concurrency doesn't cause skipping or errors if there's a logic issue with your code / mapping then its possible you could be trying to edit the same record twice somehow, but there's as much chance of that happening at concurrency 2 as 8 or even 15. and that would almost always generate a "record has changed" error

Weighing in that I have seen this occur myself, but not consistently. It only happens with two particular M/R scripts, and no errors are ever caught or logged, yet we see a small handful of records, usually 10, that just don’t process. Re-running the script they process just fine (no changes to the query or code). They definitely are not created while the script was running, they were always created at least a day or more earlier. After going back and forth over those scripts and exhausting every possibility, I ended up just adding a small query at the end of the summarize step to check for any missed/skipped records. At least, we are alerted about them and can re-run the scripts when it happens. My suspicion is that it may be other customizations deployed on those particular records that occasionally interfere.

Thanks @scottvonduhnfor the insights, would you mind sharing the sample query that you add at the end of the summarize step?

You can also send email with the JSON of your search as an attachment that you are passing to your map stage. To see whether those records were included in that search, could be a timezone thing as well if you are having filters related to date. Also if possible you can pass in the list of all the records updated to the summarized stage and send another mail for the same. And then compare both the dataset in excel. Might give you some more insights.