I think this was my problem. There were quite marginal challenges and pushes early on, but because both teams were doing it, I ended up letting them go, only giving the obvious ones whereas I should have seen this and started calling everything, no matter how marginal.