I'm curious, what's been the highest transaction v...
# general
s
I'm curious, what's been the highest transaction volume you've seen for a client or somewhere you've managed their NetSuite environment? Started a new role not too long ago and the volumes are very high compared to anything I'm seen (circa 100k transaction lines per day). It presents a whole different set of problems from what's typical so I'm interested to hear other people's experience
👍 1
a
Some examples of those problems you are having?
➕ 1
n
@Super-Kev curious to know the problems.
One problem is reaching the monthly transaction lines limit.
s
Big one is integrations hitting concurrency limits often. There's been work to batch transactions to manage the number of integrations hitting NetSuite at any one time but still happens time to time. Other one is saved search hitting timeouts, users workaround this by running multiple searches for 1 period. We're still well within our monthly transaction line limit, current tier's maximum is 50m
a
@Super-Kev Integration hitting concurrency limits is a very common issue, but it all comes down to a good integration architecture; there are multiple ways to improve that but it is complex, it would require individual evaluation of the different data flows and a re-design some of those data pipelines (integrations). The searches timing out would also need some sort of evaluation per use case etc. Both are common problems but most of the time are a design, implementation or fine tuning issue that is fixable.
People keep thinking NetSuite is just another database and when integrations are designed or implemented around that assumption these problems and issues arise.
g
We were in the 30k range of daily transaction lines although we just switched a bunch of stuff to reduce that. The 10k sublist limit can be pretty annoying. Have to work around searches and timeouts pretty often. Built a custom payment portal to get vendor paid as Electronic Bank Payments would time out. Built a report snapshot tool to generate some heavy reports as live queries would time out. All sorts of fun but no true concurrency issues yet.
a
One solution that is hard to sell because accounting and stake holders do not understand this is to design you accounting layer connected but separated from your data presentation layer. Your accounting (Invoices, transactions, etc) do not need 5k or 10k lines because the Customer want to see those on the Invoice Document they receive (that is data presentation layer) and it can be decoupled from your accounting or precess.
💯 1
Bottom line is, the Customers should not and do not define how you handle and record your accounting operations, they can request how they will see or would like to see the break down of what they owed you, but how your company GL impact is recorded is your and only your company decision and business.
s
@alien4u appreciate the responses, great insights. And agree, the company's had rapid growth and the current process/architecture was designed for a different time. I think ultimately they're going to have to only bring in a summarized level data into NetSuite (currently NetSuite almost replicates Snowflake in the level of transaction detail) One question if you don't mind, on good integration architecture. Does that mean optimizing the pipelines feeding into NetSuite or optimizing the NetSuite side of the integration? Reason I'm asking, the more I look into our current setup the more I feel there's not many options within NetSuite to do this (other than chuck more money at Oracle for more concurrency slots etc). Business logic is all handled upstream and is piped into NetSuite via it's REST APIs and all the task orchestration is handled outside of NetSuite
@Geo sounds like some similar problems our users are facing (and I'm glad to hear other people facing the same haha). The report snapshot tool intrigues me, did that come out of necessity or was it just a quality of life improvement for users?
a
@Super-Kev A good Integration Architecture includes many different considerations: • Which systems are pushing data to NetSuite and which data? ◦ Does the integration implements any sort exponential backoff algorithm? ◦ Does the integration implements a retry queue? ◦ Does the it has proper segregation of duties: ▪︎ Create Customer (one data flow) ▪︎ Create Transaction (separated data flow) ◦ Does it has proper observability (shared exceptions handling logic and logging)? • Can some portions of the integration or data flow be offloaded to NetSuite? (If NetSuite initiates the connection to an external system there is no concurrency limits restrictions imposed).
s
Hmm interesting, it's mainly Hightouch pushing transaction data from Snowflake into NetSuite. There's a small amount from a P2P system but volumes are very small But for most of those questions the answer is yes except for the exponential backoff algorithm and the last one
c
Does the transaction data have to immediately go into the transaction or can you store that data in a folder/file to get processed by an MR so that you aren't tieing up the system? Synchronous vs Asynchronous processing. Netsuite is absolutely terrible at real-time processing.
s
TL;DR nope, and it's a potential solution idea I've had
j
With that kind of volume, I typically see summary level invoices and a custom customer portal to view the source database (not NetSuite).
👍 1
a
@Super-Kev For your particular issue if you have most of what I mentioned above, the take away is: stop treating NetSuite as another database; if people in your company believe is just a database they can save a lot of money and just build a UI around Snowflake with AI in a couple of weeks, tailored to the company, problem solved.
➕ 1
n
Instead of pushing data into NetSuite, you can try pulling from NetSuite.
s
Thanks everyone, appreciate the thoughts and discussion!