I agree and would say that being locked up in one place and the way this stains your biography and thus impacts your life after prison is punishment enough, at least for us rich Westerners. (I don't wanna imply that in pre-modern times or in third world countries a decent life in prison, being warm, well-fed, secure, able to read and exercise and so on, would not have been too comfortable.)
Furthermore even a rapist or a murderer might not commit the crime again when he gets out (assuming that it isn't a life sentence). Making prison hell reduces the chances for actually becoming a better person afterwards. Not to mention that claiming to defend civilization against crime while being fine with incarcerated people getting hurt is utterly hypocritical.