The crystals referenced in "Mudd's Women" are not the paddle crystals seen later. Spock has one in the briefing room and it is a fist-sized (as per Scotty's lament about all the power channeling through a single crystal the size of his fist) clear faceted large diamond-esque crystal-ly looking crystal.
I posit that there are four of these crystals (which are not at all un-like the dilithium crystals as seen in TNG) that are used in anti-matter reactors to provide main power.
I suggest the paddles are a manufactured product. When raw, natural dilithium is faceted to work in the reactor, the ground off material is pressed into a standardized paddle form. The natural cut crystals can't be re-energized (as per Star Trek IV) but the smaller paddle crystals can be re-energized. Perhaps through use, the artificial crystalline structure is disrupted but these can be put into a re-energizing device which re-aligns the structure.
I further suggest that these paddles are used as batteries, and the battery power is in fact banks of these paddles. The times we see them being used in conjunction with the structure in the season 2-3 Engineering set could suggest that they are used (like the battery in a car) to jump start the system.
Makes sense to me...