a Map/Reduce script can process larger csv files, ...
# general
s
a Map/Reduce script can process larger csv files, though there is a 10 MB file size limitation
s
I like where you're going with this! I also imagine it would be pretty quick too... Even though the files are ~50 MB each, splitting them up into 5 different files sounds WAY better. But how would I use the CSV file to drive the map/reduce script?
Yeah, just confirmed - I would have to split each file (there are 4 of these files) into ~20 different CSVs. 80 CSVs just seems absurd.
s
basically, read the file in the getInputData phase, and pass it to the map phase in an array. you can keep each line as a string of comma separated values even, and then in the map (or reduce) phase just split the value on commas, and process away. you'll need to have a consistent order of columns, though, to make that work
s
Definitely want to explore this Map/Reduce route for sure.
Oh NICE
Yeah I see exactly what you're talking about
Okay super super thanks for this - going to look into this now! Super appreciate it.
s
Here is a basic template:
mapreducecsv.js
🔥 1
❤️ 1
💎 1
✔️ 1
👍 1
s
Oh my god Scott. I am tempted to tell you how much I love you right now. WOW. Just the quickest question though - I would set the values on the map function and end it with a context.save? Or would I declare record in the map function and then use that to set the values/save?
SRSLY thanks so much.
s
you'd probably be doing a
record.load
and then
.save()
, or possibly you could use
record.submitFields
if you are really just touching one or two columns on the sales order
within the if block of the map phase
s
Ah yeah, submitFields is def where I wanna be. You are the friggin' MAN, man.
s
I have been in almost exactly this situation before (with Invoices) so I know the pain
s
Thanks SO much for helping mitigate some of the pain! Currently having trouble triggering the submitFields or any record loading for that matter in the map phase... Did you use the reduce function at all or did you do all of your processing in the map function?
s
i was able to use the map phase for it all. in order to make it efficient, you'll need to have the internal id of the sales order. There are times when
submitFields
just doesn;t seem to do anything, but doesn;t thrown an error. Then you have to do a full
record.load()
and
salesOrderRecord.save()
after updating the necessary fields
I generally put some logging in my map phase when I first start writing it, to confirm what I am getting from getInputData: try throwing in
log.debug({ title: context.key, details: content.value });
at the beginning of the map function, and make sure you are getting what you expect
s
Yeah, I have the Internal ID and yes, I'm having that issue where submitFields is doing absolutely nothing at all... And even I have a log happen right after the submitFields that returns absolutely NOTHING. It's SUPER strange.
I'll give that a go though, you the bomb Scott!
s
Yes, it's a strange, and i've seen it more than once. My guess is that perhaps a workflow or user event script may be preventing or reverting changes made via Inline Edit (which is the context used by submitFields), but not by a regular Edit, which a record load/save does.
180 Views