you could potentially remove only THAT item. but yes, the only way to access the cart is via commerce API.