A quick idea: a cleric who saved a town from destruction became a local hero. He took up residence nearby and served as their special protector for many years. One day, he joined a band of adventurers on a quest against a lich. When he returned, he was a woman (thanks to a curse) and she remained so for the rest of her life.
Each year, on the anniversary of her return to town, all the residents dress in opposite clothing.