I have a suitelet with a checkbox. I have a client script that does something based on the checkbox. Let say the box is not checked, then the user check it. The client script does the validation and return an error, I want to undo the check from the box, I did return false but the box stay checked. What is the trick to undo ?

Generally returning false in your validateField is the way to go

Validate field will validate the field that was changed but it doesn't actually revert the value. If you want to implement that logic you would have to call setValue( ) on the field before returning false

in his specific case, its a checkbox that starts unchecked, returning false should work fine for rejecting the checked value.