But Michelle Dockery, who plays Lady Mary, reassures fans that as far as she knows, no other major characters will be leaving the show, which is set to return to PBS on Jan. 5, 2014.
"We're all doing series 5 next year," she told a TCA panel at the Beverly Hilton Hotel on Tuesday.
Dockery admits that she herself was shocked when she first heard about the third-season finale twist involving Matthew Crawley's fatal auto accident.
"My first reaction was, 'Oh crap, what is going to happen?' I thought, 'Where can the story go now,' we spent all this time having this on-off, will-they/won't-they relationship?"
But the show's writer and creator Julian Fellows persuaded Dockery to see the ending as a new beginning.
"As much as I think it was sad to see Dan [Stevens] go, it opens up an opportunity for Julian to take the character in all new directions," the actress says. And, by "new directions," she means new love interests.
"The fourth [season] is about recovery. She's starting a new life," says Dockery. "There's various suitors that are being thrown in her direction. But she's very reluctant to go through with any of it because, of course, she can't stop thinking about Matthew. It's hard to bond with the baby because, of course, she's going through the grief."
Matthew Still a PresenceAs for her late husband, "He's still very much a part of her thoughts. She'll never get over it. You never get over losing someone. It's learning to deal with it and move on. He's a hard one to replace."
Dockery explains that as Lady Mary grieves, she has trouble facing her newborn son. "You don't see much interaction between the baby and Mary," she says. "She looks at him and sees Matthew." (Dockery adds that in an aristocratic family of that era, it was customary for most of the early childcare to be done by nannies.)
Season 4 will begin in the winter of 1922. "It's very much the '20s," says Dockery. "It's very much the bright, young things. Rose, the cousin, and [sister] Edith very much represent this new wave of fashion, of dining in restaurants with a man, without being accompanied. … It feels closer to where we are today."