Wouldn't the natural outcome of that argument be that every guest star is a one-off character that never returns?
No. But it should be used wisely, not as something to simply make up for lack of imagination. There comes a point where you blow the credibility of your universe.