James Baker is accused of pointing a gun at a man who was waving a Confederate flag, along with his family, on a roadside in Chesterfield County, Virginia, on Monday.
Mark Wilson, the patriarch of the family, told WTVR: "He brandished a weapon, he chambered a round and put it to my head."
Wilson's teen son Mark Jr. added:
And the guy starts to yell at my father and get, got out of his car, and pulled the weapon and chambered a round in it and put it to his head and said, "Your cause isn't so great now, is it?"
"The police went and investigated, and they let him go. He admitted everything and they just let him go," Wilson told NBC 12.
However, Baker has not issued any statements regarding the incident.
Chesterfield Police originally didn't charge Baker, but did charge him on Tuesday with a misdemeanor for allegedly brandishing a gun, reports NBC 12.
Wilson's family will continue to publicly wave the Confederate flag, which was carried by pro-slavery forces during the Civil War.
Wilson claimed that he has generations of family who have "died for this country," but it's not clear if he was referring to the Confederate forces who fought against the United States of America.