If I were writing a SuiteScript to grab a 2,000,000 line CSV from the file cabinet and create custom records from it just a single time, would this be a Map / Reduct script or a different type? How would I trigger it to happen just once?
w
Watz
12/15/2021, 6:56 PM
Yes, I'd use a map/reduce for that. (Or a saved csv-import, but I think those are limited to 50k lines?)
Move/delete the file when all records are successfully processed.
Consider that you need to handle that some lines in the file won't be processed successfully.
Perhaps putting the error ones in a new file in another directory for errors.
t
The Usual Suspect
12/15/2021, 7:13 PM
I think you can use a file reader or something but 2 million lines might exceed some governance. I think the file module only reads files up to 10mb but that might be for non text files. I'd split up the files if you could first
e
Eric Kapalka
12/15/2021, 7:45 PM
Thanks, @Watz! In my case, the limit for CSV import is 25,000 lines, and it takes 4 hours to load that many. I actually have four of these files to load, which increases the need for a scripted solution. @The Usual Suspect I fear that that might end up happening -- I might see if a library like papaparse.js can work in NetSuite. I've used that in Node.js to deal with files that are gigabytes in size.
d
donaldtran
12/15/2021, 8:01 PM
create an on-demand map/reduce script and run it once. if you have time but just need to truly run this just once, have your on demand script run in this manner:
1. parse a line and create a custom record
2. delete that line from the csv
3. when you reach the end of governnance or whatever limit you set, have this script schedule another script into the queue if theres any csv lines remaining
4. the other script simply schedules that same M/R you ran above
b
battk
12/15/2021, 10:15 PM
papaparse is questionable for large amounts of data
battk
12/15/2021, 10:17 PM
it doesnt support the way suitescript streams, so you probably would be limited to 10 mb files unless you are willing to code a FileReader to act like a stream
battk
12/15/2021, 10:17 PM
luckily map/reduce supports csv files as the return type for getInputData, so you may not need to handle parsing