You could walk away from doing Sunday games as others have suggested but you've asked how to discharge stress after the game. I'll try not to get too deep here.
To me, stress from officiating at any game is self-generated. Unless of course you are threatened or assaulted. You either allow the moaning, whinging and verbal abuse to get to you or you don't. What you mustn't do is take it personal. Once you take it personal, you will start to get stressed. But stress that you are feeling as a result of a football game could be brought on by other things going on in your life. The feeling of stress is as a result of a chemical imbalance in your body. Your body releases Cortisol to combat stress. This can be as a result of lack of sleep, bad eating habits, lack of exercise or other things that are playing on your mind. You have to look at someone juggling plates (you may not be old enough to remember TV entertainers that used to juggle plates on various poles) You juggle 4 or 5 plates and everything is fine, add a 6th and 7th and they all come tumbling down. It's the same with stress, if you have a few things on the go, maybe struggling with sleep, maybe a few worries around work, family or college/uni then the football match could be the 6th or 7th plate that creates the stress.
The way to combat it is to create Dopamine in your body, this is the 'feel-good' hormone that combats stress. You get this when you exercise or when you receive some good news or take part in an activity that makes you feel good. You don't get this from alcohol by the way.
So, my advice would be to look at anything else that may be causing you stress first, see if you can do anything about them, check to see if you are taking the abuse as personal to you, see what your diet, sleeping and exercise regimes are like and take things from there.
2 more things, try not doing a Sunday league game but going to watch one instead. See how people react to a different referee, guaranteed it will be the same. Then finally, once your game has finished, write down flash points in your game. Could you have avoided that booking, or red card. Should you have issued a yellow or red to avoid an escalation later on? One player in particular was moaning throughout, could I have managed him differently? Could I have used the sinbin? Was there a decision that changed the way players were? I had a Sunday league game yesterday and sent a player to the sinbin after 55 seconds, a record for me. It was warranted and just plain stupid of the player. I had no dissent for the rest of the game though.
I've reflected after a game and identified a decision I made about a throw in 10 minutes before a sin bin and subsequent red card for offinabus. The players had a great sense of misjustice as a result of my decision which on reflection may have been wrong. Had I got that right then the subsequent events may have been avoided.
Once this reflection has been carried out and written down, I write another note on how to improve myself next time. I just find it helps writing it down. It also helps reading my notes before my next game as it helps to focus my mind. This helps to put that game to bed. I then go and do an activity that I enjoy and the game is forgotten about.
Might work for you, might not, I hope that some if this is helpful though.
Best of luck