See, you'll think that until something like what happened to me, happens to you.
In my single player world I've got an extensive mine cart station set up take will take me all over my map, the workings for this are all hidden underground (lots and lots of redstone and tracks going on). As it happened, when I was landscaping my gardens, a lake ended up being on top of where the works for the mine carts is. This isn't a problem at all normally, but there's a few places where there's just 1 block between the workings and the water (you can see where this is going...).
The space where the workings are used to be a sand pit that I dug out, so I walled up bits of the sides to neaten it up once I was done setting it all up. Everything inside is lit up, except those little pockets behind the walls. An enderman must have spawned in there at some point, and got out into the redstone. Then water happened at the system and destroyed it.
I only find out about this when try and go somewhere and the card doesn't arrive. No idea when it happened, but looking at the total of 4 blocks out of place it's obvious how it happened. I'm not an idiot, so I was able to find a backup that had the place intact from before the patch. But if this was on a multiplayer server I'd be insanely pissed off.
The idea of the ender men is really cool, but they're little more than griefers in practice.
EDIT: Didn't see your post before I posted. Yeah, groups of them can really the mess places up. On my multiplayer server we've got a big mansion project going (with hidden doors, electric lighting, running water, stuff like that). They're really messing it up. Not too much, but enough to annoy people. Server is set to peaceful for now as getting a group of them would terrible in there.