I have a Confederate kepi that I wear. I have ancestors who fought on the Confederate side, not because they approved of slavery (they hated it, in fact), but because they felt that the North was becoming too much like England was at the time of the Revolution, in being oppressive on matters that had nothing to do with slavery. But, people have a great capacity for seeing something and thinking that the only reason for it is what they believe....that there could not possibly be another explanation.
Yeah, sorry dude, but that's a bunch of Lost Cause Confederate apologist nonsense. I'm not saying you're a Confederate apologist, but you've unfortunately borrowed the rhetoric, hopefully unintentionally.
First off, the Southern colonies were home to the most Loyalists (irony alert) during the Revolution, hence the British Southern Strategy where they wanted to use Tories to hold onto territory gained as their army advanced North. Later, the Confederacy had no problem accepting British assistance in the form of blockade runners full of goods and in the construction of ships designed to break the naval blockade, or the British upper crust and merchant class' tacit support of the Confederacy for primarily economic reasons (while remaining outwardly neutral, and the common people were generally more supportive of the Union cause). So this idea that Southerners were so bitterly opposed to the British way of things doesn't really hold water.
Also, this idea that the South was so concerned about states' rights only applied to states' rights to continue the institution of slavery and to have it be supported even in states that had abolished slavery. They were perfectly fine with imposing on Northern states' rights with the Fugitive Slave Act of 1850, which strengthened the Fugitive Slave Act of 1793, and forced non-slave states to return escaped slaves to their masters. They did this under threat of an earlier civil war that only the Compromise of 1850 would abate. Also, the Northern states failure to support the continuation of the institution of slavery and uphold their obligations under the Fugitive Slave Acts and other laws relating to slavery were among the primary grievances cited in the Southern states declarations of secession. The states imposing their will on the others were the South on the North, under the coercive threat of civil war, not the other way around.
Every Confederate state cited the preservation of slavery as the primary motivating factor for their breaking away from the Union in their declarations of secession, so while certain individuals were more invested in preserving slavery than others, if you fought for the Confederacy, you knew the primary cause you were fighting for. So this idea that you could hate slavery but voluntarily become a Confederate soldier is a popular post-war noble Lost Cause concept that doesn't hold water. There may have been a relative few, but they are outliers. Even people who didn't own slaves profited by it and supported it because it gave them someone else to look down on and have dominance over even when they were dirt poor themselves.
What are you basing that on anyway? Do you have firsthand documentation where your ancestors write about how much they hate the institution of slavery but they'll join the slavery upholding Confederate States Army because those damn Yankees are just so oppressive about our right to oppress others? Or did you hear whitewashed stories down the years from relatives who didn't want to just come right out and say, "Your Great Great Great Grandfather Jebediah wasn't necessarily a bad person, but he fought for an evil cause."
When people see you wearing Confederate paraphernalia outside the context of a Civil War reenactment, movies/TV sets or theater performances, or history museum acts, why is the onus on them to assume non-malicious motives for wearing it? Would you assume someone walking down the street with a Nazi flag armband or a white guy with a swastika tattoo was just misunderstood? Would you feel the same way if you were Jewish, or Romani, or gay? Think about how that looks from the perspective of a black person when you wear a symbol of their ancestors kidnapping and enslavement. Why should they have to assume benign motives on your part when you're flaunting a symbol of their treatment as chattle and subhuman?
I'm not saying this because I think you're bigoted. I'm saying that you need to consider what that symbol means to others and look a little more critically on your family history. You have nothing to be ashamed of by your ancestors, since their actions were not your own, but when you take pride in the actions of those who supported a horrific cause whose primary purpose was the preservation of slavery, and which resulted in the deaths of hundreds of thousands of people, and spawned an additional century and a half of racial animus because the South couldn't accept their changed reality, and you continue to do it after being told about how doing so affects others, then yeah, that starts being problematic. Before anyone objects, I'm not saying racism is only present in the South.
Anyway, I'm just saying that maybe wearing the Confederate kepi around town is not the most considerate thing to do.