I'm sure if you thank him for the offer but say that you don't think it would look right then he will understand.
Many years ago I did a county league game at the far end of the county - 70 miles each way - with the away team local to me, and less than 5 miles from my door. I had to drive right...