SPOCK: Decoding memory banks. I'll try to interpolate. The Valiant had encountered a magnetic space storm and was being swept in this direction.
KIRK: The old impulse engines weren't strong enough.
SPOCK: Swept past this point, about a half light year out of the galaxy, they were thrown clear, turned, and headed back into the galaxy here.
Looks like they had no intention of leaving the Galaxy or even exploring the region at the Galaxy edge. They just got swept in that direction by a "storm".