https://netsuiteprofessionals.com logo
Join Slack
Powered by
Hey folks, quick question. I'm trying to use a ma...
# suitescript
m
Hey folks, quick question. I'm trying to use a map reduce script on a saved search with about 500,000 rows. The map reduce script really doesn't like this very much, and fails (probably because it gathers all the information before executing, and 500,000 lines is a lot of entries). Would I be better off exporting the search to a CSV and then pulling that into a map reduce? Or is there another approach? I know I could go back to a 2.0 Scheduled Script... but I'd prefer to stick with a 2.1 Map Reduce if possible.
j
when and how it fails? are you returning an array or a search object on the getInputData entry point?
m
@Jorge Salas Does not give a whole lot of detail about why it fails. It fails on getInputData, and I'm returning a search object.
(so technically netsuite is doing the getting of the data, not the getInputData function itself).
j
As counterintuitive as it sounds, I've consistently seen better performance running several smaller searches instead of one big one. Try splitting the search by subsidiary, location, or something similar so you end up with smaller data sets, then return the combination of all of them as an array in getInputData. you can also try delete unnecessary columns from the saved search
m
@Jorge Salas I'll see if that works. Thanks!
s
Does the search actually complete in the UI? Try execute the search in your console. Is it unexpected error you get? Try catch and log the message? Rewrite as SuiteQL?
👆 1
m
+1 to the using SuiteQL for input. Orders of magnitude faster.
☝🏻 1
m
@SimonC Search completes fine in the UI. Can also export as CSV. No error messages. Perhaps I'll try SuiteQL, had not tried that yet (too used to saved searches).
s
You could also try to run the search and map into javascript objects and pass that maybe youll also get a more explicit error if something goes wrong. Also add logs
m
@SimonC I'll look into that.
s
If you map make sure to limit the columns and props to only the ones you will use as I've seen mention of memory limitations on Reddit
m
@SimonC Yeah, it show some memory limits in the documentation. I think the primary one to be concerned with is 200MB.
👍 1
CSV export is 67 MB... so I should probably take measures to shrink it so the script doesn't need to be adjusted for a while.
To be honest... it might be best to do a CSV export and then iterate over the lines with file.getSegments or files.lines.iterator (doesn't really require more governance... probably should be fast enough to complete before timeout). Might end up doing that. (Each line should be < 10MB no problem)
s
As you wish. Does the map reduce job fail instantly or after some minutes?
m
It fails after something like 6-8 hours. I actually didn't get to see the exact time because it failed at night.
s
Oh that changes things. Just curious what are you trying to do with the data?
Also forgive the obvious, but you sure it fails at getinput stage? Strange that it takes so long to error out
m
Summarize it for an email. It's basically getting some information about the inventory valuation report, but since you can't touch reports with Suitescript, I'm recreating it as a saved search. So this search is basically an inventory history search. Anything that touches the inventory account gets included, and the values should match the inventory report for on hand and amount at any specific time cutoff. Then I'm taking that and using it to classify certain inventory into age groups, and showing the value and amount for those age groups (that's what goes in the email)
Yep, it's definitely at getinput
99% of the way there with the search (it's a little off). But the script was having troubles with so many rows. Thus why I asked here. Ironically I had originally approached the search a different way, and the script using the CSV export -> read method, but the search was having problems so I scrapped that. So I swapped to the new search method and a new script method. But it looks like the script is now the part having troubles. So now I'm going to try the new search with the old script method 😛
s
Good luck. One other idea if it doesn't work out and if you're allowed to work with the data locally just write a node/python script and call netsuite from the outside using suiteql via Rest, max page the results and process to your hearts content
m
@SimonC Could technically do that, yeah. But it's a lot easier for management if it's just built into Netsuite 😛. So hopefully I can get this working 😄 We'll see though.
b
It might not be number of rows but the amount of data in the rows in combination with the number of rows. Try making the search return only the id and then do a search per id in your map stage.
💡 2
m
@battk I'll try that here if the csv method doesn't work in a moment.
@battk @SimonC @Jorge Salas CSV export -> trigger script method worked just now. Got it down to 47 seconds executing time (29 for csv export, 18 for script), so I'll be sticking with it most likley. Thanks for all your guys help on this though.
👏 1
e
N/task/SearchTask would help you here, with 
addInboundDependency
👍 1
s
Nice
m
@ehcanadian Yeah, that's what I did 🙂. Apparently iterating through a CSV is better than MapReduce for large amounts of lines. What really surprised me though is the speed. This is WAY faster than using a traditional MapReduce script... like ridiculously faster (I was expecting processing to take hours... now it's less than a minute). I might swap some of my other MapReduce scripts over. Assuming they don't need to load records or do anything with governance during the map or reduce stage.
6 Views
Open in Slack
PreviousNext
Powered by