For me, people in any situation can show their anger towards another person (not the ref of course) without adding insults or personal jibes. So, on the FOP the use of "F*ck Off!" is perfectly adequate, but adding something referring to someone's skin colour, sexuality, appearance, etc. takes it into the realms of OFFINABUS.
Therefore, if I hear it, they will be walking. It doesn't matter if I personally find it offensive, etc., by personalising the sentence either the recipient, a teammate or spectator may be affected and we cannot be last week's ref on these sorts of things.