Interesting question. I think that both civilizations would be willing to get along despite their differences. After all, the Federation is somehow willing to be allies with the Klingons even though they still engage in warlike behavior and conquest. So I don't think they'd let a few differences of opinion with the Culture erupt into enmity.
Of course, the Culture is a far older and more powerful civilization, so they'd probably have the advantage. They're also pretty loose and inclusive for the most part. I think that over the long term, the Culture would gently assimilate the Federation, or at least portions of it. Not so much by deliberate design as through simple coexistence and shared interest, with the Culture being so loosely affiliated that anyone who's on basically friendly terms and participates in the overall community is essentially part of it, no formal political union required (at least, that's the way I seem to recall it).
As for the AIs, I don't think the Federation has any firm institutional prejudice against them; they're just fairly new, and the UFP doesn't have much experience with them. But the legal and cultural precedents so far with respect to Data and the Doctor have tended toward acceptance and inclusion. As for genetic engineering, there are lingering fears in the wake of the Eugenics Wars and the 22nd-century Augment crises, but I think enough time interacting with a culture in which genetic enhancement has not led to a race of conquerors would change public opinion.
As for Special Circumstances and intervention, maybe the Federation's involvement in the Culture might serve as a tempering influence on that intervention. I'm sure the two would debate the issue intently, and might eventually arrive at a middle ground between their approaches, as well-intentioned and mature participants in a debate often do.
So in the long term, we'd end up with a Culture that included the Federation as one of its many subcultures, and that remained largely the Culture but with some modulation toward Federation ideas, or at least inclusion of Federation ideas as part of its diversity.
Plus, Starfleet ships would end up with much more interesting names. "These are the voyages of the starship A Peculiar Variety of Diplomacy."