I'm beginning to think that @errol is onto something though. If the file streaming works as advertised, there would be no reason for the getInputData() stage to "read all the rows at once, then dole out to map() stages." Instead, it would be "read next row, kick off a map(), read next row, kick another map()" So, as long as I don't then have reduce() stages after every map() stage (which, I don't), this could be just the ticket.