We have an EUR bank account that has already 0 EUR balance. However, the USD balance, base currency, is not zero. How do we wipe this out? Through Currency Revaluation records?

you need to make a journal into elimination sub

yes your currency reval should clear the USD balance to 0