Swift, 20, has just released a new album, Speak Now, and isn't in a position to get entangled in a romance, according to a source close to the singer.
"Taylor is very serious about her career," the source tells PEOPLE. "For her, it's all about the career right now. She's not really interested in getting heavily involved with anyone."
Even after this weekend's social events – which included a visit to Saturday Night Live set and a friendly brunch in Brooklyn – the two are just hanging out for now, a Gyllenhaal source agrees.
"They're cute together," says a source close to Gyllenhaal, 29. "But I don't think there's anything to it. Everyone's talking about it. They hung out. But that's it. I don't think they're a couple."
Whether romance blossoms between the two remains to be seen, although the Swift source predicts she'll look outside the entertainment business for long-term love.
"If she does have a serious relationship, it's probably going to be with someone who's not in the industry or someone who's not a household name," the source says. "With Jake, she probably figures, why not be seen together and keep everyone guessing."
Swift's rep couldn't be reached for comment.