Yes, as fans you can absolutely be disappointed. What bugs me is this sense of entitlement that fans get about things, as if actors are indebted to them for the rest of their lives.
This. How many times does an actor HAVE to do something for the fans? Until SOME of them are satisfied? Most? All?
He did his job. He considered coming back. He decided that it wasn't for him. Disappointing, sure. But, seriously, not the end of the world.