Yep, discovered that that the hard way as well. I had to build a pretty hefty dirt wall around myself before the skeleton wake up calls stopped.
I really hope that someday the game adds some sort of lost ruins, or hard to find items scattered around the map. One shortcoming of the game is that unless you need cactus, clay, or sugarcane, theres really not much incentive to go out exploring. Just admiring the scenery doesn't seem worth the risk.