It is hard. I also found the emotional side really hard for a few years TBH.
Seems like you are very lonely out there. I learned to love refereeing through running the line. It was the better matches, camaraderie, community, learning from better refs, going through tough incidents together...