What is the best way to handle this case?... a script with multiple deployments, all accumulating certain amounts from 100K records and saving it in the same journal? Right now, they are overriding each other. Is there something similar to transactions in NS? Or do you recommend using N/cache? (which I fear a little bit)

Wouldn't optimizing the script and using a single deployment an option ? What exactly happens in each stage can you elaborate?

That's what we were doing initially, but it was taking a lot of time. The script goes over a record called shipment, and the amounts of the shipment that get calculated and added to the journal (we create one journal per customer per day) differ based on the type of customer and other factors. So what we are doing now (we actually rolled it back, but this was the idea)... is creating a deployment for customers with odd internal IDs, and another for the ones with even IDs, and we do that with shipments IDs too. But we kind of forgot that the deployments will keep overriding each other while saving to the same journal.

you haven't even told us what type of script this is? are you using a scheduled script? a map reduce?

Sorry, it is a map/reduce. (the one that creates the journals) The script fetches 400 shipments... maps them to customers, calculates the amounts, and upserts the journal. There are also around 5 scheduled scripts that usually run at the same time, some of them process new shipments (just creating them, not calculating anything), and some of them send data to the client's server. All of that usually took around 8 hours. Now the number of shipments sometimes is more than 200K per day, and the script that aggregates the amounts runs all day... and still some shipments do not get processed, and we have to change their dates to get processed the next day. (using another scheduled script)

How many journals it's creating updating ? Create/update of those journals is happening in map or reduce stage ? I feel like you are over complicating your solution. And ideally it could happen in a single map reduce script deployment and should not take longer than an hour (depending upon the volume of transaction affected and not on the number of customers you have to deal with). Unless other things are slowing the operations down.

8 hours? what? that's insane something you're doing is wildly inefficient

It is straightforward. Can I share it in private and take some of your time to take a look at it?

no thanks, its my Sunday, I'm not volunteering for work 😄

Still thankful... I will try to find the issue. ☕

wait i thought you said it only ever created 1 JE?

No, 1 journal for each customer. Customers here are the companies that our client delivers their shipments. So they used to deliver or ship around 100K shipments per day, and now it is more than the double. So 200K shipments,... I don't remember how many customers... let's say 5K... we need to process the shipments and create a journal for each customer.

why does each customer need its own JE? customer is at the line level on a JE, you can have multiple customers per JE

Would that make a difference?

I'm wondering if you could do 1 JE per shipment - with multiple customers, and then just have it be a UE on whatever this shipment record is instead of a a scheduled MR process

That's what we were doing at the beginning, but we started to hit the transaction line limit.

the transaction line limit on a JE? with one JE per shipment?

The tier lines limit. With a journal for each shipment, you get around (200K * 5) lines With a journal for each customer, you get around (5K * 5) lines

right account tiers, wasn't thinking of that

Do you still think that this is slow?

it fetched 134 shipments, and ended up upserting 62 journals, in 3 minutes

not at all, I was thinking a single write operation... you have 62 write operations, that's actually pretty performant

Great! You made me doubt everything. 😅

sorry!

No... thanks very much for your time. Hearing another opinion about it is always good.

so what are these JEs doing from a business/accounting perspective? moving funds associated with shipping costs from one account to another?

I am not aware of all of it. But it records how much money they should keep (shipping costs) and how much money they should send back to the customers. And maybe some other stuff that I am not aware of.

okay then yeah that's performing well it seems, I don't think you're gonna gain efficiency by changing the code itself. I'd want to take a step back and look more broadly at the whole process... what are we trying to do? is this approach the best way to handle it?

Yeah, exactly. But this is more of functional work than technical work. But I remember telling them at the beginning to keep these calculations on their system, and send us only the results... they refused, they want NS to do this work. When you ship something through a shipping company, the shipping company takes the money from the buyer and returns it to you. And it may cut their cost from this money, or give it all back to you, and then ask for their costs. That depends on the contract. They also issue these payouts based on the contracts. So some customers want their money the next day. And some want it once per week, etc. And there are other factors too. You need to create journals based on these factors and decide how much money should be sent to customers today. Make the back transfer. And send the information back to the client's system so the customer can see on their dashboards that their shipments got delivered, and the bank transfer was made, and it is linked to these specific shipments.

depends what you're doing... if you're querying the same thing over and over somewhere then yeah putting in cache will make the access to it much faster

that's just saying the best practice for cache is to get it from the cache but if its not in the cache, do the search and load it INTO the cache... you'd never NOT setup cache that way so its weird they express it in the help like that

depends what you're doing

Yeah, it could be useful later if I wanted to speed up some parts, but not for the solution I had in mind right now. I was thinking of relying 100% on it. Not to save the journal until all the accumulation ends... not gonna work I guess.

oh wow yeah I would only ever store search/query results in cache to make them quickly accessible, I wouldn't write and update the values to it mid process

Got it.