I'm analyzing best approaches to handle duplicate data. A little bit of context, we are receiving CSV data daily but the CSV could contain repeated data that was already processed in the previous days. The final result are sales orders/invoices created in NetSuite. How do you typically handle/identify duplicate rows? I'm thinking to create some custom records like this: • Parent Record (Job/batch/import record) to work as a container • Child Record (individual batch or group record) to have history of the data processed (here I'm grouping the sales orders/invoices by date/customer) So I'm curious how others are implemented mechanisms to make sure you are not reprocessing duplicate data and create like a unique key or something like that. Note: There are some map/reduces involved.

Use external IDs where possible.

Let me read a little bit more about it. Is the external ID something that should come with the CSV from the source of the data? or is it something I create in the script?

external id should be the primary key for the data source

It doesn't matter much as long as you're consistent. Also note, once an external id is set on a record, it can't be removed.

you can also potentially concatenate a few fields to ensure uniquness too if all you have from the source system is a simple number

Hey this is really useful guys! is there anything I should have in mind related to the fat that an external id can't not be removed? I'm assuming it cannot be edited either

They can be updated, (CSV import) you just can't clear it

Great! thanks for the clarification and your insights @ehcanadian @Anthony OConnor, I'm gonna dive deeper on this.

you can run server-side scripting w/ CSV imports, so you could also write some dupe detection logic and bail before it submits if its a dupe.

Hey @creece, just to clarify, I'm not using the native CSV Import tool here, it is an in-house importation integration using suitescript.

does the in-house solution group the data at all? sounds like that would solve it too

We are getting rows of sales and we are grouping the data in different ways (there are some settings that allow by category, service, etc) but the data does not come grouped, however I'm checking that we are recieving an ID that could be useful, probably there are more things to take into account here but my main doubt was how others were implementing mechanisms to detect duplicates. In my case I'm getting sales orders lines and then grouping in different ways based on settings.

Belt and suspenders, filter the data before processing it, and setting an external id to hopefully help with weird edge cases

you need to find some unique column in your csv to store in netsuite in a field so you can do a search/query to see if the transaction already exists