Taylor Swift and One Direction's Harry Styles: Are They Dating?

By Patrick Gomez

11/17/2012 at 10:40 PM EST

Taylor Swift appears to be taking her love life in a new direction.

The "Never Ever Getting Back Together" singer is seemingly taking her lyrics to heart as she moves on from recent ex, Conor Kennedy, and enjoys the company of One Direction hottie Harry Styles.

"I had to literally do a double-take," an onlooker tells PEOPLE of finding Styles, 18, with Swift, 22, on the set of The X Factor Thursday morning.

Styles was on hand to watch Swift rehearse the debut of "State of Grace," which she performed later that night on the Fox reality show.

PHOTOS: Taylor Swift: Singer, Actress, Photographer?

"He was smiling at her while she rehearsed. When she was done he jumped up on stage, picked her up, put her over his shoulder and carried her off stage," the onlooker says. "The whole crew was really surprised."

The young singers were also spotted by X Factor host Mario Lopez, who says he was slapped on the back by Styles during Swift's rehearsal.

"I said, 'What are you doing here,' " Lopez said on his 104.3 MY FM radio show Friday. "And he sort of [pointed] toward Taylor."

Lopez went on to say he later saw the two "hand-in-hand."

A telling sign of the budding relationship may have been a look Styles shared with his bandmate Niall Horan a week earlier after Horan told PEOPLE his favorite song of 2012 was Swift's "Never Ever Getting Back Together."

When asked if he would ever date Swift, Horan gave a small laugh, looked at Styles and answered with a succinct, "no."