I think it was everyone on the Valiant--it wasn't so much a matter of having time-traveled as it was that the ship itself was the eye of the storm or whatever. Basically the whole time-reversal thing went on around it, but the ship itself and everyone on it had actually experienced The Year That Wasn't and therefore remembered. (And yeah, Martha's parents definitely did--Mrs. Jones was originally going to shoot Saxon for it, remember, saying that "it still happened," and the fact that the whole family had gone through all this was part of Martha's reason for leaving the Doctor.)