Thomas W. Dixon's The Clansman assumed Lydia Smith was Stevens' (called Stoneman in the novel) mistress. Given strong religious values, being a secret mistress may have been very offensive to Smith. And having a severe club foot (and any other health problems or impairments) could have left Stevens convinced he was unmarriageable. Though it is certainly possible that they were a couple, it is equally possible they were not. Really the only argument that they were is that Stevens could have had anyone for a housekeeper rather than tolerate the whispers. Given that Stevens was as irascible and stubborn as depicted this is not conclusive.
The point is that the reading of the Thirteenth Amendment was a major use of artistic license. This reveal was the climax of Stevens' subplot (or "arc" if you insist on that term,) even if the climax of the amendment plot as such was, as was correctly stated, the bells ringing. The issue could be rephrased, was it good drama to give Thaddeus Stevens so much of Lincoln?