For actual record load times, I don't imagine there's any appreciable difference at any sort of realistic-sized scale. If you need to apply certain logic to these fields to get certain results, that could be very different. E.g. running the "contains" operation on a large number of text fields can start to get pretty resource intensive. But honestly it's going to be scripts, workflows, and custom fields utilizing searches on said records that are going to be the things slowing stuff down that you actually have control over versus the type of field. Just use the type of field that makes the most sense/best fits the data and use case.