I've recently been on the other end of this, where my brother and his wife loaned me a large amount of money a few months ago. I had an agreement drawn up even though he was really chill about everything and I've already repaid it. Personally, I don't think you should ever loan money, you should only gift it. Family member or not. If someone is in the position of needing cash, you don't really know when they will be in the position to repay it, if at all.
In this specific situation you have to choose between the money and your relationship. If you want the money, you can consult a lawyer but it sounds like you didn't even have an agreement so you probably won't get what you're after. If you want the relationship, make nice with him and drop seeking interest. That seems like a dick move anyway if you never talked about that before.
This whole way of thinking is a bit weird to me honestly, but I think I just have a very different relationship with my siblings.