Most of the really brutal Saviors were already dealt with. Negen's in prison, Simon's been killed and his zombie is chained to Sanctuary's fence. Likewise, Simon's loyalists who were with him when he killed the Heapsters were all killed. Dwight's in exile, with possibly everyone else believing he's been executed. The only one of Negen's top Lieutenants who is still around is Regina, who aside from attempting to kill Eugene in response to his betrayal has never seemed particularly evil. I guess there's also Laura, though it was her idea to surrender to Rick and the Gang, and so far she seems to be making a genuine effort to establishing smooth relations. Indeed, Daryl actually apologised after he snapped at her in the premiere, so she must be doing something right. Even among the rank and file Saviors, the only one who truly seemed a genuine threat last season was that one who had a grudge with Morgan, and he's dead now.