Is this an instance where you're sending out orders to each of these addresses one time maybe?
If so, I'd definitely adjust your automation (or whatever) to use the custom option on a transaction level so each address is only present on the transaction and doesn't save back to the customer. Assuming you don't need it on the customer for some reason.
As for what to do about the ones already with that many address records, oof. Assuming you don't truly need most of them, you could maybe kill off most of them with a CSV import to replace sublists and then only add back the ones your really need?
If those are truly all address you need, I've seen people have more success using subcustomers to represent a Ship To. Still messy, but easier to report on and much less clunky. I think there's a limit of 10k subcustomers per parent though, so if this is going to continue, you'd have to be cognizant of that.