There are certain stumbling blocks to solve for cryogenic propellant depots, but it is unquestionable that it is going to happen real soon. We won't be able to afford going anywhere without it - even with "HLLV's".
Anyone expressing "doubts about depots" is heavily invested in creating huge launch vehicles we cannot afford to operate. I would express huge doubts about their objectivity. It's like arguing back in 1900 that gas stations were a bad idea - and they we should do all transportation with huge trucks that had to fill up at the refinery.
We don't need 100 mt launchers. Not even 70 mt. The solar system is open to us, with with even smaller launchers, such as FH. Even the Delta IV-Heavy (at 25 mt) could be used for lunar flights if we wanted to - only 3 launches per mission. And with the FH @ 50 mt, one launch is enough to send a spacecraft with crew or to resupply a Lagrange or lunar orbit station.
And until we have cryogenic depots, we can still do hypergolic depots. (which will certainly be needed for a reusable lunar lander) As sojourner states, we already have a hypergolic depot in orbit - ISS. We can build far better ones if we just fund it.