@Jacob D I had to solve a similar issue with Map/Reduce in the past, in my case, parsing a giant CSV file using MR (think hundreds of thousands of rows.) The few things I recall learning:
* No, N/session is not available when the MR script is run as administrator.
* N/cache has a gigantic caveat that, the cache can expire at any random point in time. I found this to be documented, and also saw it empirically. There IS an N/cache function which allows you to "re-load" the cache when it expires which is highly recommended by the documentation. Some combination of N/file and N/cache could be an option there.
* The "trick" I found worked best; pass the desired persistent values as a script parameter at the point in time that you execute the script. Every stage of the script can see script parameters, and you don't have the crazy cache expiration issues.
... In my case, the "script parameter" trick worked well, as, I only had a couple-hundred-byte string to pass around. Your case may be different though, since you have thousands of internal IDs. I could see you needing to first create a file, and then pass that file ID as a script parameter.