Martok readily adopted a non-relative into his House, establishing at least that Houses go beyond the nuclear family, and possibly suggesting that they are more like family businesses than families: trusted non-relatives may be in a majority, really. And I'm sure that Kurn would have accumulated at least some manpower around himself even before going public with representing a surviving House of Mogh (in his big brother's absence).
How "mobile" would these non-relatives be? Are they ultimately just paid or unpaid labor that will shift alliances when a House is dishonored, loses its head, or simply falls on hard times? Or do they stick to the House to the bitter end? The former would make it easier for Houses to "fall" with such totality as suggested in the dialogue. And really, a House operating multiple starships, estates and businesses would need lots and lots of trusted members, way beyond a literal "band of brothers" (or sisters) and well into the "my adopted brother's cousin knows the man who hired this warrior" territory.
Some argue that "Houses" are just for the noble, upper class Klingons (think Kor vs. Martok).
...Yet Martok did have a House! And it was his in name, not Sirella's, FWIW - although it might still have been something Martok got from his wife.
Dialogue from "Soldiers of the Empire" suggests that Klingon ship crews aren't used to being led by warriors who don't have their own House (or who don't belong to a House). This might mean that all skippers are House members - or that all skippers of Imperial ships are House members, while private vessels may be owned by Houses and commanded by House members, or owned by non-House parties and commanded by such.
Certainly Worf didn't automatically assume he would be of the House of Martok simply through being Martok's underling aboard the Rotarran
. An employee apparently needs a special invitation - an adoption rather than a mere contract. Although again we may argue the Rotarran
was being operated as an Imperial ship, just with the House of Martok providing both the ship and her commander, and not as a House ship; in the latter case, everybody aboard might have been considered a House member.