"Night Terrors" some good, scary moments in that. But I think the prize goes to "Schisms." Really good, creepy, eerie episode that makes the "alien abduction" story work in this setting. The recreation of the abduction room on the holodeck is a great, fantastic and chilling scene.
According to Frakes, he was actually pulled into that muck.
For going into it/coming out of it, sure. But I highly doubt that's his face (with open mouth) coming out of the stuff, especially considering -according to Frakes- it was made from Medamucil and printer's ink.