Trying to retrieve 165K records from a NetSuite restlet that is sending it in JSON format - receiving gateway timeout error. Any one know how to fix this other than reducing data size?

We could maybe provide some performance suggestions if we saw the restlet code, but off the top, I can't think of any way a single restlet call could return 165k records of any size

had a feeling! 😞 so we’ll have to return pages and then query back for more pages.

that's generally how these things are done 🙂

@leo Yes, you'd have to paginate 😉

That, or if possible, refactor such that NetSuite can work on the full result set in the background and push the response back to you, e.g. via Map/Reduce

I’m not a developer guys so this is helpful. Thank you 🙂

A proper RESTful API (end point) and this includes RESTlets should have some sort of pagination implemented, so you can get for example 1000 results per request and you call that RESTlet 165 times.

not that i dont think the suggestions relating to implementing paging are incorrect

The response JSON for 165k records and the 10MB response size limit means... you only get ~ 60 bytes of data per row... that's not much, if you're literally just bringing back an id you might be ok, but anything more than that and you're gonna hit the 10MB limit even if you're getting the data back in time.

didn't NS introduce some streaming output capability that is not subject to the 10MB limit?

I know they did for files... I'm not aware of a similar thing for RESTlet responses

I'd cache the data into several JSON files (each capped at a certain number of records) and how "real time" you need... Then retrieving those files based on a predefined naming convention, join them, then parse them.