Taylor Swift and Harry Styles
Janet Mayer/Splash News Online; Don Arnold/Wireimage
11/17/2012 at 10:00 PM EST
appears to be taking her love life
in a new direction
The "Never Ever Getting Back Together" singer
is seemingly taking her lyrics to heart as she moves on from recent ex, Conor Kennedy
, and enjoys the company of One Direction hottie Harry Styles.
"I had to literally do a double-take," an onlooker tells PEOPLE of finding Styles, 18, with Swift, 22, on the set of The X Factor
Styles was on hand to watch Swift rehearse the debut of "State of Grace," which she performed later that night on the Fox reality show.
"He was smiling at her while she rehearsed. When she was done he jumped up on stage, picked her up, put her over his shoulder and carried her off stage," the onlooker says. "The whole crew was really surprised."
The young singers were also spotted by X Factor
host Mario Lopez, who says he was slapped on the back by Styles during Swift's rehearsal.
"I said, 'What are you doing here,' " Lopez said on his 104.3 MY FM radio show Friday. "And he sort of [pointed] toward Taylor."
Lopez went on to say he later saw the two "hand-in-hand."
A telling sign of the budding relationship may have been a look Styles shared with his bandmate Niall Horan a week earlier after Horan told PEOPLE his favorite song of 2012 was Swift's "Never Ever Getting Back Together."
When asked if he would ever date Swift, Horan gave a small laugh, looked at Styles and answered with a succinct, "no."