The only advise I'd give on this is to try when and only when you are ready, and aim to have a colleague with you.
Balancing refereeing with life doesn't get easier, wife, 3 teenagers and animals, just remember that for all but a few it's a hobby that puts a bit of cash in your pocket.