I will have to start importing like 50k records a ...
# generals
Simon
09/14/2023, 8:50 PMI will have to start importing like 50k records a day. It will take forever. Is there's a tool to do it ? Can Celigo do it ?
r
rico
09/14/2023, 8:51 PMYup. Where is the data coming from?
a
AK47
09/14/2023, 8:52 PMwhat records? sales orders?
s
Simon
09/14/2023, 8:53 PMData are coming from a txt file. We are connecting to Amazon and extracting the data
It's a custom records.
a
AK47
09/14/2023, 8:54 PM50k is 2 csv imports. import to custom record is usually pretty fast. Can certainly use celigo...or simply build an import in netsuite, but native csv import could work too
s
Simon
09/14/2023, 8:56 PMCSV import is limited to 50k~ lines. Cannot be used
a
AK47
09/14/2023, 8:56 PM25k, so two imports gets you to 50. If you have suitecloud+, you can have two imports simultaneously, but i agree...its not ideal
r
rico
09/14/2023, 8:56 PMWhat are you pulling from Amazon ? Would it make sense to integrate from Amazon to NetSuite without the export?
You can use the data loader with the free Celigo account if you are just doing the files uploads.
a
AK47
09/14/2023, 8:57 PMCeligo has integration with amazon, so as orders are placed, they would come in.
AK47
09/14/2023, 8:57 PMbut if you want to use the csv and have a human do the import...it can be built as a suitelet in netsuite
s
Simon
09/14/2023, 8:59 PMI should not have said Amazon, it's not relevant. We have a file or 2 everyday to import, each with 50k~ records.
I need a solution to import quickly.
Importing 50K records in Mysql, take about 50 seconds. I'm looking for a solution like that
a
AK47
09/14/2023, 9:04 PMimport to custom records is quick, but i don't expect it to be that quick
t
Tyler Lamparter
09/15/2023, 7:00 PMLike @AK47 said, NetSuite won't be that quick no matter what and a simple customer record would for sure be the fastest. You can either use NetSuite's csv uploader, but it's limited to 25k rows per file so you would have to chunk up the file each week. You could also use Celigo and there are a couple options. We have a data loader (https://docs.celigo.com/hc/en-us/articles/226949488-Create-a-Data-Loader-flow) which is free and it would let you just select the whole csv and not have to chunk it up. This still requires you to manually upload it each week. The other option would to automate it in someway by using normal Celigo flows and to pull the file from S3, ftp, mysql, or whatever. This would require a celigo license, but you then wouldn't have to touch it each week.
As for performance, we would process the file up to specified concurrency limit you set on the connection in Celigo. For example, if you only let us use a concurrency of 5, that's how fast we would go. If you are on a low NetSuite tier or don't have SuiteCloud+ licenses added, you may only have 10-15 concurrency max anyways which would the limit for any solution.
👍 1