Is there anyway I can update a custom address record? For example:

rec = record.load({
    type: 'salesorder',
    id: 13087989,
    isDynamic: true
})
shippingAddress = rec.getSubrecord({ fieldId: 'shippingaddress' })
shippingAddress.setValue({ fieldId: 'addressee', value: 'XYZ' })
shippingAddress.save()  // this throws an error
rec .save() // this updates the address on sales order by creating a new custom address record. here the old nkey is not equal to the new nkey of the shippingaddress subrecord.

You have to use shippingAddress.commit(); for the subrecord.

I don't think commit is doing anything in this case. It's still creating a new custom address record, which has a different nkey. basically doing shippingAddress.commit() and then rec.save() is same as rec.save() without doing the commit first.