It depends where you are refereeing. For a few years I always turned up in a suit and tie, wherever I was refereeing but then changed that.
Saturday afternoon Supply - suit and tie
Saturday afternoon league - suit and tie
Sunday morning - tracksuit and kit underneath
County FA games - suit and tie*
Academy games - suit and tie
Semi finals and Finals - suit and tie
*except for junior games in which case it is tracksuit unless semi final or final