the problem is i have a very large set of results i’m processing and each result may require significant governance units. apparently map/reduce has a hard per result cap on governance and it won’t yield automatically if you exceed it. it’ll just throw an error. it only yields if governance is exceeded for the overall phase

what kind of hardcore processing are you doing that requires over 5000 points per result

structured right, you could potentially give each "result" quite a lot of governance units. For example, there is no reason that you are limited to just one Reduce phase per result. You could do some pre-processing of the result in the Map phase, and for each unit of work, create a Reduce context to deal with it. I have done this before, using complex keys (some id or value unique to each result + a sequence letter, for example). You might need conditional logic in the Reduce phase to perform different work depending upon the sequence letter in the key, but it is doable. However, if all of the processing for each result has to be performed sequentially, then it won't help, as you can't guarantee the ordering of the reduce phases.

Ok, thanks for the suggestion. Basically, i’m loading an advanced promotion discount search, going through each result, and if the result uses an item saved search for eligible items, i load that search and store the item names from the search into an array. the problem is there are potentially millions of items that could be returned for one item search per result of the promo discount search

how many advanced promotion discount are there

ok. maybe i’ll try that. thanks!

global variables wont work (at all honestly)

yeah i was having trouble retrieving the output property. i’m new to map/reduce, so i’m sure i’m probably doing something wrong. can i pass stuff from one phase into the summary context and access through the output property?

you can only pass the keys and values you write

i think i couldn’t log it at all

keep in mind that a lot of the stuff in ss2 use getter and setter functions

i’ve got a more serious problem at the moment. my reduce phase timed out because i guess the result was too big. i thought i might run into this, which is why i was trying to figure out how to yield in a scheduled script using 2.0. during the reduce phase, if my promo object has an item saved search, i’m running the search and storing the results in an array. but some of these searches pull 2-3 million results. once you hit that per result cap with a map/reduce, the script just fails

well, there is no yielding in ss2 scheduled scripts

*can’t

i still say do the promo search in getInputData and have it get all the searches you have to run

can getInput handle that much though? the logic from top down is i’m running a promo search that may have around 100 results, push the fields i need into an object, checking if it’s got an item saved search, if it does, push all the items into an array and store in the original object, then at the end push all the results into an array and create a json file

100 searches at 5 points each is 500 points

right. but what if one promo result has an item search that contains a few million

bad math

no?

ok. so you’re saying get pagedData objects for all my item searches during the getInput phase, then fetch the results in another phase?

correct

gotcha. ok, i may try that. thanks!

you would need to split the data so that each key processed by a map would represent one page of data

fyi i don’t think getting the paged data for the item searches ahead of time during getInput will work, because when you pass that data into the other phases and it’s stringified, you can reference the object directly to get the actual page ranges, but methods like fetch won’t work anymore to retrieve fields and such

plan on getting a new PagedData each time and using the stringified pageRange to tell which index to fetch

Yeah i don’t think map/reduce is going to work. The reduce phase has a 5k per result limit. Let’s say all i’m doing in that phase is fetching all my paged data. Some of these results may have 2-3 million lines in the search. Take 2.5 million as an example … at 1k lines per page that’s 2,500 pages i’ll have to fetch. A single fetch() is 5 units. That means to fetch 2,500 pages it will cost me 12.5k units, far exceeding the reduce phase limit. I’m thinking i may have to script this in 1.0

ive been trying to steer you to fetch individual pages in the map phase instead of getting all pages in the reduce phase specifically to avoid that problem

Yeah but the map phase has a significantly lower limit, only 1k

individual pages