Is a mass/reduce script faster than a scheduled script even if we only have 2 processors?
t
texmc
11/27/2019, 4:22 PM
if you make both processors run in parallel in an MR then yes
e
ericbirdsall
11/27/2019, 4:25 PM
Which I can do on the M/R deployment, correct?
r
reptar
11/27/2019, 4:28 PM
how you handle map and reduce determines how many processors/threads will be used if i'm not mistaken. concurrency would affect this if there were multiple instances of the script running at the same time.
t
texmc
11/27/2019, 4:32 PM
correct you can do it on the deployment. note you need to manage the processors and the schedules to make sure they are not busy and scripts end up waiting for a long time
s
stalbert
11/27/2019, 4:44 PM
I'd argue that measuring would be the best way to tell which is faster.
e
ericbirdsall
11/27/2019, 4:46 PM
@stalbert fair enough - The situation is, the scheduled script is already written and runs properly. I didn't want to write a M/R script if there wasn't going to be a time savings. But I guess ultimately, I just need to do it
s
stalbert
11/27/2019, 4:47 PM
empirical evidence is best in a situation like this. I don't know how complex it is but hopefully not too big a deal to convert the scheduled to a MR script?