It seems like they don't really make that much of a difference unless you're planning on min/maxing your civ. Otherwise, like others have said you can just offset the difference using policies.
I'm starting to get a "meh" feeling from this game. Too often I search for an option only to find that it's been removed from this game Apple style, to protect you from yourself apparently. So instead of being able to set my auto workers to only work on unworked tiles my choices are to let them run rampant or micromanage. The game seems to have been dumbed down by several degrees.