Hey smart people, I have a question about a Map Reduce. I have a list of about 70,000 things that I need to compare nightly with another list of about 1000 things, using some fuzzy string matching logic. The second list (of 1000 things) is a record type that has a couple different fields to compare against the bigger list. I’m thinking of using a MR to go through each entry in the big list, but would rather only retrieve the smaller list once (rather than retrieve it from database each time for each of the 70,000 things). Is there a way to do this, get that smaller list once and just pass it around the MR? I suppose I could dump it into every entry of the key-value pairs but that seems….extreme.

you could maybe use the cache module. I've never used it for anything like a list of 1000 things so I'm not sure what limitations it might have, but that's the first thing I'd look into.

Can't we use a global variable in the map reduce script?

nope, if you have a global it will only be available in the GIS and the summary stages, not the map or reduce stages

I’m doing all the searches w/SuiteQL

its been a while, but if i remember rightly there's limits on the # of keys you have in a cache, but you can do something really dumb like nest it, and you can have as many as you want... something goofy like that i remeber, NS may have "fixed" it

make sure to also use the loader function when getting the cache so that it would repopulate the cache if ever it returns null. Also take note of the max size limit(500kb) of the cache you are putting.

Thanks guys

My experience is that a single map instance shares the global variables with each iteration until it yields. Load the list in the beginning of the Map stage and it can probably be used many times until it yields for out of governance points/time. Regarding fuzzy search, we've been using string similarity with good results.

...if you're loading the list at the beginning of the map stage, and there's 70,000 map stages, you're going to load the list 70,000 times?

no

how do i write code to execute in the map instance, but not for each key-value pair?

We have this function in a map/reduce.

fetchEmployees()

is called in the beginning of the map-function and then we access the EMPLOYEE_LIST throughout the map-instance.

const EMPLOYEE_LIST = {
    fetched: false,
    employees : [],
}
const fetchEmployees = () => {
    if(!EMPLOYEE_LIST.fetched) {
        search.create({
            type: search.Type.EMPLOYEE,
            filters:[ ['isinactive', <http://search.Operator.IS|search.Operator.IS>, 'F'] ],
            columns: [ EMPLOYEE.FIELDS.ENTITYID ],
        }).run().each(result => {
            const entityId = result.getValue(EMPLOYEE.FIELDS.ENTITYID) as string
            EMPLOYEE_LIST.employees.push({
                id: result.id,
                [EMPLOYEE.FIELDS.ENTITYID]: entityId.toLowerCase(),
            })
            return true
        })
        EMPLOYEE_LIST.fetched = true
    }
}

so your

EMPLOYEE_LIST

declaration is outside the map? and when a new instance is created it will reset the global to empty?

Yes

right, what I mean is, the declaration will be ran again in the instance...

Yes, once per instance.

good to know, presumably that works in the reduce stage too

I'd assume so