Hey, has anyone successfully implemented a data ex...
# general
a
Hey, has anyone successfully implemented a data export service to export all Netsuite data into a Data Warehouse such as BigQuery? Did you use an off-the-shelf solution or built something in-house? What were the rough cost estimates for the data transfer process? We're having significant difficulty coming up with an optimal solution. Thanks!
m
We've done something using SuiteAnalytics Connect (JDBC) to Snowflake using integrator.io (Celigo), happy to chat.
a
@michoel thanks, I have a list of all the off-the-shelf stuff already. But we want to implement something ourselves, as the cost is 2 orders of magnitude less 😆
Anyway, for reference for anyone looking into this in the future. Google Cloud Data Fusion is almost there, you can use the JDBC connector by setting up a new Connection via Database (NOT Oracle/anything else). BUUT, you can't actually use it in the "Replicate" accelerator to load only changed rows. You could implement a CDC solution in a JAR file and use a custom plugin, but the further away I can be from Java/Oracle the happier I am 🤢 Another option is to build a custom program/script/whatever, that connects via ODBC (because ew Java), loads the data, and then pushes it to BigQuery, and running it in a Cloud Function for example (or CKE, Cloud Run, my grandma's computer, etc.)
s
I've recently integrated direct connection from NetSuite to Snowflake to query data. We are also working on a solution using Python for data extraction.
🙌 1
extracting data might be also possible using SuiteScript but considering the load, it might be better done outside.
a
Yeah, and today's proprietary technologies are tomorrow's legacy systems 😆. We're considering Python+Polars ourselves. Connector-X would've been a great option, but their ODBC connector has been WIP for 4 years 😆