Is it possible to remove the Leave button?

use a workflow to remove the button

I don't think you can deploy workflows to the duplicate records page?

Thanks