Stars Tony Goldwyn and Kerry Washington, who play the fictional U.S. president and his sometimes lover, respectively, were joined by costars Guillermo Diaz, Katie Lowes, Jeff Perry, Bellamy Young and others – and Scandal creator Shonda Rhimes – at 1600 Pennsylvania Avenue.
"This is what's known as an epic picture," Goldwyn captioned a Friday Instagram photo of him, Rhimes and Young in front of the White House.
But that wasn't the only behind-the-scenes image Goldwyn, 55, shared from the cast's tour.
The cast was also invited for Friday's concert on the grounds in celebration of International Jazz Day (including Aretha Franklin's tribute to Prince, with "Purple Rain").
"It was amazing," Lowes raved to PEOPLE at Saturday's Garden Brunch hosted by Tammy Haddad. "POTUS and FLOTUS walked in and sat three to four rows ahead of us. We're overlooking the Truman Balcony, they made the Truman Balcony go purple when Aretha Franklin started covering 'Purple Rain' and asked the entire audience to sing along with her."
"I was bawling crying," Lowes said. "It was surreal."
The actress added that the Scandal cast is all looking forward to attending the main event Saturday night. "I'm excited for Barack to do his thing," she said. "Since it's his last year he can kind of leave it all on the dance floor, can't he?"
In anticipation of the big night, Washington, 39, shared a few photos on her Instagram Saturday. "#Olitz in the real rose garden," she captioned a shot of her and Goldwyn.
Like Goldwyn, she also posed with a portrait of Democratic presidential front-runner Hillary Clinton.
"I am so excited about Saturday, I mean it will be bittersweet, but we are all so excited to see what [President Barack Obama] says," Washington said. "He is a brilliant and hilarious man and we always get to see him be brilliant but we don't always get to see him be so funny."
Washington's on-screen, on-and-off love interest Foley told PEOPLE just hours before the White House dinner that while his last time at the event was "sort of an intense, over-the-top experience" he's feeling more relaxed this time around – not that he's not still excited.
"I think if you're not excited to see the president in the same room as you there's something wrong with you," he joked. "I'm always excited to see Barack Obama."
"My cast, I love hanging out with my cast," he said. "We're all going to be together and enjoying each other off-set so that's a good thing."
"We're going to take this damn thing over, watch out."
Reporting by TIERNEY MCAFEE