Taylor Swift and Conor Kennedy
Cindy Ord/Getty; Douglas Healey/Polaris
08/01/2012 AT 04:15 PM EDT
Fresh into his relationship with Taylor Swift
, has Conor Kennedy already joined the ranks of her famous ex-boyfriends – including John Mayer
, Taylor Lautner
and Joe Jonas
– whose memories with the country crooner play out in her music
Not so fast, according to one source, who says this love song is still being written, with Swift falling hard for the son of Robert Kennedy Jr. and the late Mary Kennedy
"They have been dating for a bit," says the source. "She is swept off her feet."
Age is clearly not a factor – he's 18 and she's 22 – but what may be is the Grammy winner's affection for his extended family. Even before they were spotted getting cozy
, Swift made no secret of her fascination with the Kennedys.
She attended the Sundance Film Festival last January to take in his aunt Rory Kennedy's HBO documentary, Ethel
, about her mother (and Conor's grandmother).
"She really just loves my mother, which is so sweet," Rory Kennedy told ABC News
some time ago. "I'm a huge fan of hers – I have young girls and we had been to a few of her concerts."
"I met with her at one of them, and she had read a ton of books about our family," Rory continued. "She was inspired and wrote a song."
And Swift said in an interview in February's Vogue
: "The only time in my life I have ever been star-struck was meeting Caroline and Ethel Kennedy.
"I got to spend the afternoon with Ethel a couple of weeks ago," she added. "She is one of my favorites because you look back at the pictures of her and Bobby, and they always look like they are having the most fun out of everybody. You know, eleven kids, all these exotic animals on their property."
She went on to say, "I've read a lot about them."