Hull design, for one. The lack of a reinforced sail meant Nautilus could not safely break through pack ice in the North Atlantic or the Arctic oceans (where attack subs were expected to operate during the cold war) and the more traditional hull shape meant Nautilus' highest velocities could only be achieved while surfaced. Later designs also reduced from 8 to 6 and then to 4 the number of torpedo tubes, redesigned the attack center, and incorporated new sonar technologies Nautilus simply couldn't equip because it was shaped the wrong way. Also, the two-screw design of the Nautilus made the ship difficult to control at high speeds.
It was a good ship, to be sure, but almost nothing about it was optimized for use as a nuclear submarine. They might have called it "U.S.S. Learning Curve."
You don't "prototype" naval vessels...
Timo Saloniemi

Well yes, actually you do, as in the aforementioned
U.S.S. Nautilus.
Nautilus wasn't a prototype, though. It was an actual active naval vessel that incorporated a lot of new technologies. The limitations mentioned above are a consequence of the fact that nobody had ever used those technologies together and thus had no idea how well they would work in practice.
USS Midway had similar issues when it first entered service, and the Navy's many attempts to correct its problems either made them worse or replaced them with NEW problems. A similar thing happened when the Essex class carriers introduced angled flight decks for jet aircraft; it took them four tries to come up with a configuration for the hangar elevators that didn't convert the carriers into floating death traps (and later they had to redesign the elevators altogether when USS Forrestall burst into flames).
Basically: you DON'T prototype naval vessels, you just redesign the next ship based on what went wrong with the current version.