To be honest, if people didn't keep pointing it out, I'd have forgotten he was in Fires of Pompeii. Torchwood:CoE is something I'd remember simply because it was such a long, memorable role, but Fires of Pompeii was a relatively brief role in a forgettable episode. Fires of Pompeii, imo, needs no explanation. CoE also needs no explanation because it's a different show, so reusing actors is more forgivable.
However, I'm all for anything done well, so, as long as it's done well, I don't care.