Once you said it I got wondering. Perhaps they use glue to fuse them to the fabric, so its part of the tailoring. Although if you were relieved of duty you had to remove it so I'm back to square one.
Glue or stitch a magnet to the inside of the uniform? So it is removable but involves a strong magnet so it won't fall off while wrestling with the enemy. Yet if you're fired you can take it off.