@David B @battk That ^^^ solves schedules running out of concurrency or governance but not parallel issues (No Deployment Available)...

Doesn't the system get littered with deployments?

Yes, it will depend how many and when do you execute them, I have seem over 400 without any issues. But again everything boils down to the use case.

400 on a single script?

Yep...

I have a script with 10 deployments which models threading, but 400, wow.

Again, it boils down to use cases... for example if a customer have Sales Orders with multiple(many fulfillments) and they want something to update fulfillments body fields when the Sales Order (createdfrom) body fields are updated then you spawn a Map Reduce every time any of those fields are updated in a Sales Order to update all the Fulfillments related to that Sales Order, if the customer updates 300 Sales Orders(CSV) for example you may end up with that much deployments or similar. this is just a random example...

Why not set a checkbox on the sales order that indicates that it needs to have its fulfillments updated. Then attempt to start the MR-script, if it is already running do nothing. Then in the summary of the MR, reschedule itself if needed. Shouldnt the map and reduce stages give you enough parallelism?

Agree, that may work and it could be potentially better but we as developers do not dictate customer requirements in that scenario the fulfillments related to Sales Order number 200 in the list may take 15min or more to be updated and the customer may want those changes instantly…again, that was just a random example there is many more where you need to trigger multiple MR tasks and creating then manually is not feasible. In the other hand if you initial map reduce is running and from that very same map reduce you try to create another task, the deployment is still in use when you try to spawn the task from summary.

The MR can reschedule itself using the same deployment. Only the task is created after the script is completed. Yeah, I understand that you want it to be processed as fast as possible, but you'll run into processor limitation if you create a bunch of tasks anyway. And why would one users request to update go before an earlier one? How many MR scripts can be processed at a time? I know individual instances of input, map, reduce, shuffle and sunmrize are limited by the number of processors you have. But can you have 40 mr-scripts running with only 5 processors? And each instance of map being executed in a FIFO fashion spread across all 40.