Hello People, I have a map reduce script that creates and merges a bunch of pdf and store them in file cabinet. The pdf creation process is happening in map stage and merging of pdfs is in reduce stage. Here is the thing, I now want to upload these merged pdf files on SFTP. What is more efficient? To create a SFTP connection in getInputData stage and pass the connection Object to map and reduce once, upload files to SFTP in reduce OR create connection with SFTP in reduce stage? Creating connection first means that it is possible that map stage takes a little too much time for execution and I have a risk of connection time out. Creating connection in reduce stage means that I'll be creating connection for almost 700-900 times. Which approach is better?

I would be very surprised if the 2 stages persisted the connection. There's probably no guarantee that the next stage even runs on the save server/pod, so the socket would not persist. I personally have written something similar, and I created a completely separate process to do all of the file uploads & had that run after the first M/R

I'd tend to agree with @verikott and take the separate process approach

That's very interesting @verikott Thank You

wouldnt matter, the sftp connection that is used to check the credentials is not the same sftp connection used to do any of the operations

@battk can you please elaborate?

a ssh connection needs to be created and later closed to use sftp

so I would need to create connection each time I am uploading a single file?

keep in mind, despite the similar name, a ssh connection is not the same as a sftp.Connection