You can wear glasses and referee, but it will cause you lots of problems, both from a dissent and practical point of view. (no pun intended :))
My eyesight is not particularly bad in strength terms (so I'm told), but I'm blind without glasses or contacts. Contacts make life so much easier all...