I agree very much, a referee is making it a lot harder on him/herself taking the arrogant stance. However, if I was to give a penalty in which both teams disagreed on my decision, I would offer my apologies, and go with your advice: tell the players to do what they feel is right.