anyway to let the system allow me to set a item rate on a SO for 5 decimals...$500.111112

You would need to apply custom logic via script or potentially workflow. You could round or truncate as needed.