True, we may be insignificant in terms of how much of the universe we can influence, but that doesn't mean that aliens wouldn't come over. For all we know, there may be species that travel around the galaxy looking for other life, just because they want to see other life.
But the problem is energy expenditure and the shear amount of time it would take for them to get here. If we imagine that there is another intelligent life-form 50 light-years away and they receive one of the SETI signals we sent out and decide to come and visit, they might be looking at a 150 year round-trip. They would most likely have to stock the ship with enough food, fuel and various other supplies to last for over a century because there might not be a place to resupply on the way.
Would all that time and energy really be worth it so that they could come here to learn about American Idol and our petty nationalistic struggles? And what if they got here to find out we died 20 years prior in an horrific war? Unless our planet really does have the Ark of the Forerunners buried somewhere under the surface then I don't think we should expect aliens to take enough of an interest to bother visiting us.