I figured it out. The pagenumber element reserves a few digits worth of space so that it's always got enough room. I guess it'll just have to be off center.