Should have been ordered off the field of play at the first stoppage then? Or not even allowed to start?
Otherwise the interpretation you are putting forward suggests that a player can opt to take the caution and do nothing about their inappropriate attire/equipment and thats the end of the matter.
Becomes more challenging when it's the GK as you have made them leave the field of play, you can't really restart until that team has a GK in place. So if the GK is being really obstinate and refusing to remove the offending item, you simply don't restart until they do, give them a "reasonable" amount of time to comply (or provide a substitute GK), then move towards an abandonment.
Or....caution before the KO for the refusal to comply, order them from the field of play at the first stoppage, if they still refuse, 2nd caution and red. Forces the team to provide a new GK and the game can continue.
Good luck selling those decisions, especially in a cup final.