The major downside is lending books - I tried it once. I was supposed to get the book back after 14 days but it has instead been locked in digital limbo. Amazon hasn't resolved it.
This completely freaks me out. That some third party can control your ownership or use rights to some entertainment you paid money for and actually keep you from using it.
I know someone who flew from Canada to Europe and travelled through a couple of different countries. They had loaded a few ebooks on their iPhone to read while they were abroad, and as soon as they stepped off the plane and switched the phone off of Airplane mode, a number of their books were locked right away due to the publisher in the country they had landed in that published the books not being the same publisher as here in Canada, and that foreign publisher had activated the security lockout for all editions but there edition. And as far as I know they still have not been able to get the books unlocked, even though they have been back in Canada for a while.
That's pretty much equivalent to having the publisher waiting at the airport to confiscate your books. Who would think THAT made any sense?