I have a RESTlet, it's doing too much, causing the...
# suitescript
n
I have a RESTlet, it's doing too much, causing the system calling the RESTlet to detect a time-out. The script is not timing out but it is on occasion taking 30 seconds to run. I tried creating a single custom record in the RESTlet to hold the data that's passed in instead and attached a UE script with all the logic in it and that runs AFTER SUBMIT. However the RESTlet still takes too long as it's waiting for the UE to complete... despite it being in after submit (did not know that). So then I had a "bright idea" to wrap my record.create in an async function and in my REStlet I call that. This does not help. 😞 So, am I barking up the wrong tree, is trying to run functions asynchronously on a serverside script (without awaiting), always going to run as if I am awaitingignore the async declaration?
Copy code
const post = (data) => {
            let nd = new Date();
            log.debug({title: "In Time", details: nd.toISOString()});
            async function doStuff(data){
                try {
                    let fieldData = JSON.stringify(data);
                    let adbRec = record.create({type:'customrecord_3en_absence_data_bus'});
                    adbRec.setValue({fieldId: 'custrecord_adb_data', value: fieldData});
                    adbRec.save();
                    let od = new Date();
                    log.debug({title: "Record Id time", details: od.toISOString()});
                } catch (e) {
                    let errorString = e.message || JSON.stringify(e) || 'No Meaningful Message';
                    log.error({ title: 'Oops!', details: errorString });
                }
            }
            doStuff(data);
            let od = new Date();
            log.debug({title: "Out Time", details: od.toISOString()});
            return true
}
Or, looking at what I've done have I made a rookie mistake? I confess writing asynchronous functions is not something I do with any regularity so it's entirely possible I'm missing some salient point.
l
What I usually do in these cases is creating a Map/Reduce to process this staging record. So it will just save the record then M/R read and process it
e
My async-fu is also minimal, but could you use
adbRec.save.promise()
instead, and not wait for it to complete? I have also seen and built what Luiz says as well; Restlets just ingest a staging record, and a M/R regularly processes them.
n
save.promise() is client script only 😞
e
I see
n
I was trying to avoid a batch process as I need the underlying functionality to be as real-time as possible. It's creating time-off request records as requested by the external system. The creation and tracking success or failure is all done in a SuiteLet and the external system updated with said info and the users new balance of leave. As NS creates the time-off change records in the background and that impacts on the resulting table of leave types I am having to take an opening balance form the employee record sublist and then do the math based on those records that were successfully created. Originally I just had a RESTlet that called a Suitelet passing chunks of data as an array of promises and awaited the response which worked beautifully, it's all down to NS being so damn slow to reply that's causing me grief.
a
you don't have to schedule the MR to run every hour, you can can use an MR with no schedule and just use the task module to run it
n
and if the queue is full?
yeah it's going to start stacking up, right?
😞
If a supervisor goes in to the system and starts banging through approved leave they might fire in 10 requests in the space of a few minutes and there are several supervisors and over 600 employees 😞
m
@NElliott You async function isn't going to do what you want it to as you need to return a promise, I'd also not define the function in your post function. const post = (data) => doStuff(data).then(result => log.debug('test', result)) const doStuff = (data) => { return new Promise((resolve, reject) => [function code - return resolve(success) or reject(fail code) on failure]
n
Hold on what is my post supposed to be in this scenario? ^^
Ah
a
10 requests a minute sounds... not at all like a volume that should be problematic?
n
Calling n/task to create a map/reduce task 10 times in the space of a few minutes would likely fill the job queue. is what I'm thinking 😉
a
dont schedule a new one if its already running
m
Post will be your normal entry point for the restLet. I would just define the doStuff function outside of the post function. The doStuff function won't exist and has to be redefined each time the post function runs. It'll be a small difference in efficiency, but there will be some.
a
have it do a search for the staging records, dont pass in the id, and it will get any/all that have come in just a checkbox on the staging record saying its processed afterwards
you can also still have a scheduled version that runs every 15m as well
n
@Anthony OConnor Thanks for responding and your suggestion. I may be forced down that route if I cannot get this server response time down but I'd really prefer not to.
s
Why don't you just get the calling side to wait longer than 30 seconds?
n
There's a cost involved in AWS with holding the connection open which will be costly over time.
s
well the fastest option then is to just 'queue' the data as mentioned and immediately invoke the background script. I would use a scheduled script here rather than M/R because it seems to launch faster in my qualitative experience and the logic you're doing probably doesn't need the M/R model if you were previously just doing it in the restlet. YMMV
2
You could also reverse the flow - have AWS send to an AWS queue then pull from that queue from suitescript. That would minimize AWS connection time (I presume sending to an AWS queue would be faster than making NS act like a queue).
n
That's an interesting idea, I'll ask the AWS guys what they think 🙂 cheers.
a
umm that would minimize the AWS connection time, but you'd still have to pull the data on some kind of schedule, I'm not aware of anyway to effective poll something from NS other than 15min schedule, so that means things aren't realtime
... if you want to make it a bunch of work for everyone you could do both!... instead of them sending you the entire payload in a post, they just send some identifier to the payload, and then once they trigger your restlest with that id you make a GET for the full data payload, which you then process on your time and they're not waiting on you to close the connection
you could also try to optimize whatever it is you're actually doing with the data on the NS to make that quicker so it doesn't hit the timeout... that's a whole other thing though (FWIW you might try turning off logging, or setting it to audit+ on the deployment to speed things up, its overlooked but it can make a real difference in time sensitive situations)
n
The code I wrote is running extremely fast, what I'm finding is, NS does not seem to be returning from the RESTlet immediately! I'm on a 1GB connection, wired, and pretty sure it's nothing on my side or the code. There's this kind of "air-gap" between the code finishing (as I see a timestamp in the logs) and it being seen in AWS in their logs. I had removed all but 2 log lines, a timestamp at the top and one at the bottom but thanks for the suggestion. Incidentally, turning the log level up a notch or more doesn't help as I recall @erictgrubaugh mentioned anecdotally awhile ago. It seems no matter the level NS makes a server request and ascertains whether it should log it or not. (unless I misunderstood the conversation!)
👍 1
e
You understood my understanding correctly 😄
🤭 1
f
I'd construct and decorate the objects outside of Netsuite, and then push them in in batches using SOAP disabling any server side processing. You can keep battling inside of Netsuite, or build yourself a real tool. SOAP is terse, but it is robust and FAF. For example, send a minimal set of parameters to an external gateway, then use SuiteQL through a rest interface to get the data that you want, then do your logic, and send back the packet to another api that then posts the data to Netsuite. This provides layers of isolation for testing purposes. With your tests written, you can easily now swap out different API endpoints and validate compatibility between versions. That's my approach, it's a bit of a sledgehammer to tap a nail, but it works.
s
I have to disagree with Fred on multiple levels. Please don't do what they are suggesting. Regarding logging - that's another reason to use NFT - it supports multiple loggers and setting verbosity really does avoid native
log()
calls if the logs are below the loglevel threshold. Furthermore, for client scripts, it automatically logs to the console by default, making no round-trip calls to NS for logging.