Just recently changed the status of some Purchase Orders from CLOSED to REJECTED by changing the approvalstatus field to REJECTED. These changes sent email messages to the vendors of those POs. Where is this email-sending step configured? I'd like to avoid this for REJECTED Purchase Orders.

Check on the Vendor record, Preferences and uncheck the email checkbox. This will stop all automated emails being sent to the vendor though, and you would have to manually send them.

I still want to keep using the automated for other instances. I just need it to be turned off for REJECTED POs.

Try the Communication sub tab on the PO ?

Nothing specific there. I don't know what to do in communication tab. I'm not familiar, do you happen to know what to do exactly?