"There are no signs of a romance between them," says the source.
Speculation of a romantic relationship first began when Affleck, 43, and Miller, 33, were spotted holding hands while filming in Los Angeles on Monday, Dec. 7. But while the two may play love interests in the film, they're not in a real-life relationship, two sources say.
"The paparazzi are there every day they are filming," a second source on set tells PEOPLE. "These photos are often used out of context. He is a great director who likes to hang out on set with those he is directing."
The first source adds, "He has had some special scenes with Sienna, including kissing scenes. They seem to get along great, but Ben has acted very professional."
Filming for Live By Night began in October and has seen the cast in Georgia, Boston and now L.A. Affleck, who split from Jennifer Garner in June, serves as screenwriter, director and actor on the project.
And according to the first source, the Oscar winner has been enjoying his time working on his latest directorial effort.
RELATED: Ben Affleck's Changing Looks!
"Ben seems very happy to film and direct," says the source. "He is in the best mood. He is very chatty and friendly with everyone on the set. He makes sure to create a fun and positive work environment"
The period drama takes place in the 1920s and tells the story of a police captain's son who becomes a powerful player in the criminal underworld.
Also starring Zoë Saldana, Elle Fanning and Chris Cooper, Live By Night is set to hit theaters in 2017.
Reporting by LIZ MCNEIL