05/03/2014 at 08:00 AM EDT
The cards were in Ben Affleck
's favor one too many times.
The actor was banned from playing blackjack at the Hard Rock Hotel & Casino in Las Vegas on Tuesday night, a source confirms to PEOPLE.
After multiple wins, security caught Affleck, 41, counting cards and reportedly swarmed the table, according to TMZ
"The security said, 'We have to stop you because you're too good,' " the source tells PEOPLE.
Despite reports, Affleck was only banned from the blackjack table and not the entire casino. "The hotel was really nice and told him he could play other games," the source adds.
In a statement issued on Friday night, the establishment said, "Mr. Affleck, a valued guest of Hard Rock Hotel & Casino, is not banned from our property and is welcome back any time."
Affleck was reportedly in the city for a quick romantic getaway with his wife, Jennifer Garner
, before he heads off to Detroit, Michigan, to begin filming as Batman
in the Man of Steel
Reporting by LIZ McNEIL and MARK GRAY