Anybody who thinks refs aren't athletes...

I've never thought of referees as athletes, but that's maybe because I'm going by the British definition:
Athlete: a person who takes part in competitive track and field events.
I think many, many referees (especially the better ones) are highly athletic but that doesn't make them athletes for us Brits.

I see however that the US definition is a bit broader and carries the more general sense of:
a person who is proficient in sports and other forms of physical exercise.
I guess this is just another case illustrating the (supposedly) Shavian observation that, "The English and the Americans are two peoples divided by a common language."