There are two things for me here, and I'm writing this with the view of someone who sits on disciplinary hearings.
Absolutely right to get rid of the manager. He will be charged and he doesn't have a leg to stand on if he appeals.
Where there is an issue is the abandonment. Where any game is abandoned due to the conduct of participants there MUST be a disciplinary hearing to determine the outcome, leagues are not allowed to deal with it until county have done this. This hearing will in effect be determining one thing - did the actions of the manager cause the game to be abandoned. Had he refused to leave the area then I would agree that he had, but based on what has been written it will be very difficult to agree that he actually caused the game to be abandoned, and rather the referee chose to abandon.
This is what all referees should be aware of, if you abandon there will be a hearing that you will have to turn up to. So if you are able to it is far easier in the long run to get rid of the person(s) causing problems and continue the game. Yes, there will of course be occasions where you have no choice but to abandon, but to me this doesn't sound like one of them. Referees that I've mentored have been mortified when they find out they have to go to a hearing, but that is the harsh reality of it - you will be sitting with the idiots that you have had to deal with and in more cases than not they will spin a pack of lies about what happened. And the commission will want to know why you didn't just send the manager off and carry on with the game.