Like I said, B.J, I think it would be really cool if there'd be a castle jutting out from the cliff face
Feel free to remove part of the wall in order to make it easier to build anything you want. What you have now looks like it could be the entrance with stairs leading up to it.