https://netsuiteprofessionals.com logo
Join Slack
Powered by
<@UEZ4CRBST> You can manage files larger than that...
# general
m
@Michael Mascitto You can manage files larger than that if they are new line deliminated. This is SuiteScript 2.0 only though. See the file.appendLines() funciton of N/File. https://STICKYOURACCOUNTIDHERE.app.netsuite.com/app/help/helpcenter.nl?fid=section_4205693274.html
m
Thanks! The problem i had with 2.0 is you can’t control yielding. And with processing a data set as large as this is you have to have control over that. Map/reduce won’t work because it only yields after each phase’s function invocation. It doesn’t yield within a phase if you’re dealing with a really big result. So i switched to 1.0, but now i’m dealing with the large file creation problem!
m
Yeah, that's a problem. Hmm... perhaps what you could do (it'd be a bit complex) is use the SS 1.0 Scheduled Script to create a few files and then use a SS 2.0 Script to merge them. That'd probably be a bit of a pain, but I think it'd work.
m
Hm that’s possible. Maybe i could make a suitescript 2.0 custom module that just handles the file creation and appending?
i can then invoke it in my 1.0 script
m
That might work.
You'll probably be working a lot with scheduled scripts I guess
Sorry that's such a pain
m
hmm although i don’t know how to add modules in 1.0 … i know you put them in your initial ‘define’ declaration in 2.0
m
You don't add them. They are in the global state
Have you used jquery before?
m
yes
m
If so, you know how you add the script tag
The scripts are imported like that
m
i see. i don’t have as much experience with 1.0, so that’s helpful
m
Yeah, if you run into any trouble, message me
It's not too bad
m
now are you suggesting to create the JSON file initially, then as i’m processing the data, append the file gradually? based on the documentation, it doesn’t look like a single append can be greater than 10 MB
so it would have to be in bites
if so, my script runtime is going to be crazy long. with the amount of data this client needs to process, it’s already taking about a day and a half to run. to load a file and append it is going to increase that for sure. i might be able to mitigate it a little by only appending after so many thousand results, not at the end of each loop. not ideal, but i don’t think there is really another option
m
Actually what I'd suggest is the following:
1) Use Suitescript 1.0 to deal with the yielding part of your script. Write the results to several files. For example, Result_1.txt, Result_2.txt, Results_3.txt.
2) Schedule a SS 2.0 Scheduled Script to read the several files and join them together into 1 file. Then delete the several files
(or something like that)
And yeah, append many lines at once if you can
it'll work better 😃
Also, while I haven't used it, SS 2.0 does have a file streaming option that might work for joining files. No idea though.
m
ok thanks!
i was just thinking, if i use 2.0 to join the separate files, won’t i just run into the same problem? during the consolidation process, won’t i have to use the file create API to create a new, combined file and the aggregated content will exceed the 10 MB limit?
m
The AppendLines() method only has the 10MB limit per line.
Which means you can work with much larger files
In addition, you can reuse the first Results file and just keep appending stuff to it
so that'll speed things up
So file.load() the first file and then file.appendLine() the rest to it
Open in Slack
PreviousNext
Powered by