The approach I take with this has worked well over the years for me and not caused any problems for me (famous last words) for an clear and obvious back pass where no ambiguity exists, I won't interfere, and if the @keeper handles the ball there will be zero surprise when the whistle goes.
Where I will shout "Not a back pass", or alternatively "Back Pass, no hands", is where there could be a question over was it / wasn't it. E.g defender slices the ball, defender tackles and the ball rolls back to the keeper, ball played off a shin or knee (or i do deem it a deliberate pass back); that way everyone is clear before the keeper handles it that I'm not going to be awarding a free kick for handling the ball, so leaves the players limited options for appealing the decision.
If a player does ask why I was "helping the opponent", and the same would be true for "hands down", "don't back in" and other go-to proactive verbal warnings and managements , a simple "you'll get the same courtesy" has always sufficed.