could you just parse the CSV file and import it as...
# suitescript
s
could you just parse the CSV file and import it as part of the script logic? it may be simpler.
b
I hadn't thought of that though there might be governance concerns based on the file size. That was one of the reasons I'd gone with a CSV import. I'll evaluate that though.
s
how many records are to imported in the CSV file?
if it's very many, I have used a Map/Reduce script for this exact scenario. You can perform pre-import steps in the getInputData stage, do the record create/update in either Map or Reduce, and any post-import steps in the summarize stage
b
It will vary from a small handful to hundreds. I don't think it will go to thousands anytime soon.
Hmm. I haven't dealt with map/reduce yet but might have to look at that.
Thanks for the ideas
s
It's well suited to this kind of job. The tricky part is learning what to pass between stages, and how the context key and values work. The keys and values are always strings.
getInputData can pass an array of strings to the map stage. The lines of CSV file could be the source of that array
this would be the basic skeleton of the Map/Reduce
define([], function() { cconst csvColumns = { FIELD_1: 0, FIELD_2: 1 }; function getInputData() { // put before import code here const csv = 'Field 1,Field 2\nValue A,Value B\nValue C,Value D'; return csv.match(/[^\r\n]+/g).slice(1); // slice(1) removes the header line } function map(context) { const csvLine = context.values; const values = csvLine.split(','); const field1 = values[csvColumns.FIELD_1]; const field2 = values[csvColumns.FIELD_2]; } function summarize(summary) { // put after import code here } return { getInputData: getInputData, map: map, summarize: summarize }; });
k
@scottvonduhn Map/Reduce Can execute big size of CSV Data and add logic in getInPutData to parse and return array of data in order to push into Map Stage or you can parse data in Map Stage As well
I would suggest you to upload file into File cabinet and load file and contents of it and Use file module to iterate through contents
s
getInputData can definitely handle parsing a good size file. Keep in mind, it should be breaking the file into separate tasks. The map stage may need to further parse the line, to get distinct values for processing.
there is a 10 MB file size limit
k
Methods that load content in memory, such as File.getContents(), have a 10 MB size limit. This limit does not apply when content is streamed, such as when File.save() is called.
b
And in my case the file is in the file cabinet and all indications are it will be way less than 10 MB
s
perfect. you can just use the N/file module to load it and then break it up for the Map or Reduce stage
@kevin I haven't tried using file streaming within the map or reduce stages. is there a particular file streaming call that synchronizes across multiple instances of map or reduce?
or do you mean iterating through it in the getInputData to build up an array to return?
👆 1
there does need to be a way to detect and remove the header line, which you don't want to process
k
You don’t need to build up Array but It’s up to you, how you wanna pass data into rest of stages
s
right, but if you don't, how do you eliminate the header line?
i suppose some logic could be added to the map stage to look for the header values and do nothing if it sees them
k
//Skip the first line (CSV header) iterator.each(function () {return false;}); iterator.each(function (line) { // This function updates the total by // adding the amount on each line to it var lineValues = line.value.split(‘,’); var lineAmount = parseFloat(lineValues[1]); if (!lineAmount) throw error.create({ name: ‘INVALID_INVOICE_FILE’, message: ‘Invoice file contained non-numeric value for total: ’ + lineValues[1] }); total += lineAmount; return true; });
Use first iteration and return false in order to skip headers
s
excellent, thanks for the tip!
👍 1
k
I have done Map/reduce many times on CSV file, especially, when you’re migrating Data from other systems, like Salesforce….
b
This is looking like a better solution. Thanks for the discussion!
👍 1