Hi folks, I am trying to pull all of my company’s data from netsuite to dump it into a data warehouse. Once I do one full pull, then I will do incremental pulls for just data from the last week or so, but I’m having issues with the first pull. Does anyone have experience pulling all data out of netsuite?

Hey @Josephine Douglas - so yes on experience. How are you doing the data pull? Via ODBC, SOAP, REST, Saved Search, etc? And how is it failing? (timeout - errors - other?)

@Josephine Douglas do you have a template for the data you are trying to pull?

I have been using ODBC (or JDBC? I’m unclear on the ODBC/JDBC relationship in netsuite. I am using JdbcHook in Airflow), but one table (transactionLines) is just timing out after about an hour. I have tried to paginate the data like so:

SELECT * FROM (SELECT ROWNUM as row, * FROM (select * from {table}
        )) WHERE row > {current_count} AND row <= {current_count+batch_size}

and for this particular table and it still does not succeed. Even if batch_size=1, the query does not seem to ever return. The table has a hundred or so columns and an unknown number of rows.

You need to specify the columns for some tables like this or it will time out due to all the joins. So basically you can't use SELECT *

Ohh interesting. I will check that out. Thank you @dynamicl!

Yes @dynamicl and @Josephine Douglas - directly accessing the NetSuite database is fairly limited. (as are most operational systems) We've implemented a low cost ELT tool to help people like you get over the data hump... https://www.distilleddata.io/services - Feel free to check it out. We're so confident this tool will help, you can sign up for a free 2 week, no obligation - proof of concept.