Hello,
I need to generate a large chunk of data fr...
# suitescriptj
j.j
12/18/2023, 12:44 PMHello,
I need to generate a large chunk of data from multiple saved searches and loading of records and then store it in a single pdf/excel file.
The issue is if I use a Map/Reduce then it will yield multiple times and how can I continue to append to the same file created earlier during the start of the job.
How can I continue writing data from the each loaded record to this same file.
Map/Reduce will surely run multiple times over and afresh, how can I share the file ID parameter to the same script run so that 1 map/reduce produces 1 single file.
Basically I need to control in a way that each yield I know the data reached and remaining to be processes and write to the same file created during first run.
Any thoughts or suggestions are most welcome.
Note - I have done this with scheduled script in the past successfully but just want to know if map/reduce can do it better or if thats the only way.
v
verikott
12/18/2023, 1:03 PMSchedule the next execution from summary, and pass the fileid in as a deployment parameter?
j
j.j
12/18/2023, 3:24 PMno that wont work as all processing is done in either map or reduce stages which will yield multiple times over, it wont reach summary stage until all processing is completed
b
battk
12/18/2023, 6:10 PMpdf and excel are both weird options for building up a file incrementally
battk
12/18/2023, 6:12 PMbut if you meant csv, the standard flow is to write each line you want at the end of the map or reduce
battk
12/18/2023, 6:12 PMthat you later combine into one file during the summary
battk
12/18/2023, 6:14 PMif you want to be weird, you can incrementally build the file in the map or reduce stages, but thats going to encounter concurrency issues if you have more than one processor running on it
battk
12/18/2023, 6:15 PMif you really wanted that route, then a script parameter is the normal place to put the file id
battk
12/18/2023, 6:17 PMthere are more clever things like searching for the file or assuming a file path if you really wanted to avoid the script parameter
j
j.j
12/19/2023, 2:18 AMthanks @battk csv or excel anything is fine, yes we need to write to the file incrementally and collect all data in a single file for reporting, seems scheduled script would be better bet while running theough the loop and rescheduling with file id and last record as parameters and append to the same file. dont need parallel processing.
2 Views