How do we clear the External IDs assigned to records? I tried doing a CSV import where External ID is just blank but it didn't work.

Did you tick the "Override current value" (or some such, I can't remember exactly) on the "Import options" screen?

On the import options screen where you choose Add, Update or Add or Update, you need to choose Overwrite Missing Fields. That works for normal fields. I don't recall ever trying to blank out External ID so I'm not 100% sure it would work for that. Note I would not have any other fields mapped when you do this. Otherwise you might accidentally blank out other fields.

best you can do is to change it to another value, no unsetting external ids

It really seems to be not possible as mentioned by @battk. Thank you.