Is there a way to get money back without also getting inventory back? For example, if product is damaged, but the customer is refunded.

Yes do RMA and when you do item receipt just untick do not stock and stock will be written off and you'll be able to refund the RMA.