O'Brien's rank is kind of a mess, and I don't think there's much point in trying to make sense of it. Riker apparently addressed him as "Lieutenant" in an early TNG appearance. For much of his time on the show, he had two full pips, and his uniform was basically identical to Worf's (Personally, I think engineering and security should've had different colors, but that's another matter).
When Sergey Rohzhenko comes aboard, he somehow recognizes O'Brien as a non-com, despite his son (whom they talk about being an officer in that very scene) having the exact same rank insignia.
Still, O'Brien's uniform stays the same until season 6's "Realm of Fear", when he goes from two full pips to one dark one, presumably to make it clear that Barclay (whom O'Brien interacts with quite a bit in the episode) is a superior officer.
Supposedly, O'Brien got some sort of promotion when he moved to DS9: In an early ep, he offers to take a transfer, since his wife obviously hates the place, but she says he'd have to give up his promotion.
Then in the 4th season, O'Brien trades his one dark pip for this weird little square thing I don't think we've seen on anyone else. Whether that was another promotion, or just some uniform change, no one knows.
BTW, is it me or does it seem like the officer to crewman ratio on starships is a lot higher than it is on a real navy ship?