Stewart and Pattinson at the Teen Choice Awards July 22
Kevin Mazur/TCA 2012/Wireimage
07/26/2012 AT 12:30 PM EDT
has packed his bags.
Following Kristen Stewart
's public apology for having a fling
with her Snow White and the Huntsman director Rupert Sanders
, the 26-year-old actor has left the Los Angeles home he shared with Stewart, PEOPLE has confirmed.
"I'm not sure they'll be able to recover from this," a source says of the couple of three years, who met making the Twilight
In seclusion and not in contact with Stewart, 22, the British star "is heartbroken and angry," says the source.
A "devastated" Stewart, meanwhile, described Pattinson
as "the person I love and respect the most" in her apology following revelations of her indiscretion with Sanders, 41, a married father of two.
"Kristen really loves Rob more than anything," says an insider. "He's all that matters to her right now."