In my experience removing permissions/lowering them auto removes the affiliated links

It will remove it from the ‘quick view’ but not the ‘home links’