I only wish they could bring Daredevil, Spidey, Fantastic Four, & X-men back to the fold of a shared universe. I miss those guys.
I feel X-Men is such a robust cast of characters that cross-overs tend to be distracting. You end up with mutants as a product of evolution and all the allegories that come with it and, oh by the way, there are other super heroes that are artificial and have none of those things but can still pose the same threats to other people. It undercuts the themes a bit. So I'm fine with X-Men being a separate universe.
Similarly, I feel that Daredevil and Punisher (might be missing someone) would work great in the same universe as Spiderman, but you don't necessarily need them in the Avengers universe.