"Insanity" is a skewed perception of reality or morality and objectively irrational behavior. Plenty of insane people are perfectly in control, they just see the world in a way that is unrecognizable to the rest of us.If you're clinically insane, you have no control over your mental faculties. Both sociopaths and psychopaths retain control of their own minds and are therefore not, by definition, insane.
As far as we know, the twins are the first example of Lannister incest. Unless you count cousins - which you shouldn't - as studies suggest it has little effect on the kind of genetic abnormalities that arise from the offspring of siblings or parents and children unless done consistently for generations without any introduction of outside genes.Keep in mind that we're not just talking about one generation of incest. We're talking about many generations of inbreeding over centuries. That will lead to a number of genetic defects. Regardless, the track record is not good for the Targaryens and Lannisters.
I would be inclined to recommend some requirement that the royals marry commoners, but, given the situation with Jamie and Cersei, that might not be a good enough solution.