That said - you can do a CSV import with the "overwrite sub lists" option to true and upload 1 address. It will remove all the addresses affiliated with it.