"That is just annoying. People say I'm always dating so and so, and sometimes it's three people in one day," says the actor, 39.
"I'm trying to make movies, work hard and do my best. Then this kind of thing happens and people start to associate me more with that. It gets ridiculous."
Aside from those pesky rumors, Butler – who also stars with Heigl" in The Ugly Truth, opening July 24 – has nothing to complain about when it comes to his current gig.
Aniston Is 'Classy'"She is one classy lady," he says of Aniston. "Everyday I go to work with her I'm always surprised about how cool, easy and down to earth and real she is."
He adds, "I'm in New York right now filming and I'm in heaven. I can't always say that about movies, but to be filming this romantic comedy with Jennifer and a story that made my side split when I read it – I'm lucky. I'm happy as a pig in s---, as they say."
As far as Aniston's current love life goes – a source close to the actress, 40, says she's still happily talking to Bradley Cooper. But when it comes to working with Butler, "They get along really well, but Jen only likes him as a friend," says a source. "She can't see herself dating him."
Reporting by PERNILLA CEDENHEIM