You seem to be failing to grasp the depths of Modell's shit-kickery. Back when the NFL was gearing up for expansion and Baltimore, led by Bob Tisch, was leading the pack, Modell talked him out of the bid. And then when Al Lerner, who was one of Modell's closest friends, stepped up for the Baltimore bid, Modell then cast his vote for Jacksonville to get the expansion franchise. Why did he fuck his friend over? Because he was already negotiating behind closed doors to move to Baltimore, and was ginning up bullshit in Cleveland (like claiming there was no running water on the upper deck of Cleveland Municipal Stadium, which was bullshit -- he was pissing and moaning over the Jacobs Field project because he wasn't going to own the new stadium).
The stadium thing is really a microcosm of douchebaggery. In addition to the sports franchises, he also owned Stadium Corporation. Stadium Corporation fucked around with the Indians while they were sharing the stadium -- Stadium would book concerts and not fix the wrecked field, would refuse to upgrade facilities, forced the Indians to appeal to the city for a tax hike for a new stadium ... guh.
And there was the little thing about him insisting that the Ravens would inherit the Browns' history, which the NFL initially agreed to until the rest of the world told them to fuck off. The book False Start, by Terry Pluto, does a great job of illustrating just how Modell and the NFL set up the new Browns to fail after the very public black eye they got from the Baltimore debacle.
He wasn't George Preston Marshall, who was pure, concentrated evil, but he was a piece of shit.