haul down one's colors
haul down one's colors or strike one's colors {v. phr.}
1. To pull down a flag, showing you are beaten and want to stop fighting.
After a long battle, the pirate captain hauled down his colors.
2. To admit you are beaten; say you want to quit.
After losing two sets of tennis, Tom hauled down his color.
Categories:
Source: A Dictionary of American Idioms