There's also this bit from "Peak Performance":
LAFORGE: The hard part's going to be calibrating the thermal curve necessary to start a controlled reaction.
RIKER: Assuming you can, can you regulate the reaction?
WESLEY: There's just enough crystal to do it. We plan to channel the reaction through the chips.
LAFORGE: Are we good?
RIKER: You're better than good. Great. Brilliant. It's going to be fun. Carry on.
That seems to suggest that the function of the dilithium crystals is to somehow regulate the matter/antimatter reaction, rather than to provide the fuel for it.