They aren't harboring any prejudice, and frankly I don't care for your suggestion that I
do. I think that was a rather unnecessary jab.
Now on to substance.
It can become a very awkward thing for a male-female pair when the speculations and rumors start flying about a friendship, when people allege something else is there and it isn't. It would be the same thing in this case--the orientation wouldn't be an issue to them or anyone else, but rather the simple problem of people assuming something that wasn't true.
That's something I've gotten a lot over the course of my own lifetime because most of my closest friends have been male. And it gets very tiring, very annoying after awhile to constantly be mistaken for a couple because of a close friendship. It's as if some people don't know how to process a relationship where there might be assumed to be the possibility of sexual interest, without assuming it is
I hated that.
But as to why Sam and Mendral would be hesitant about letting other people see too much of their brotherhood, besides what I suggested above, I think in the end it most importantly boils down to the fact that they both feel very much like outsiders.