According to the Welsh actor – who is PEOPLE's Sexiest Man of the Week! – his girls couldn't be less interested to watch their dad "on the telly," as he so charmingly puts it.
"My kids are the least impressed about any work that I do, which is quite amusing," Ellis, 37, tells PEOPLE. "Time and time again, if I'm on the telly, I'll go: 'Look, it's Dad on the telly,' and they'll sort of look up ... give it a cursory glance and then carry on with what they're doing 'cause it really doesn't faze them at all."
"The only thing they really, really liked me doing is a very gentle sitcom that I did back in the U.K.," he adds, smiling. "So yeah, they are my biggest fans as a dad but not as an actor."
Plus, there's one thing Ellis' toughest critics – Marnie, 3, and Florence, 7, with ex-wife Tamzin Outhwaite and Nora, 9, from a previous relationship – definitely don't approve of.
"What they don't like me doing is talking in any sort of characters' voices while I'm at home – it really doesn't sit well with them," Ellis says with a laugh.
And while his daughters might not be tuning in to Lucifer on Monday nights, it's safe to say Ellis has amassed quite a few admirers since the supernatural series premiered in January. (In case you're not caught up, the show follows the story of the devil, who decides he doesn't want to be the devil anymore and instead moves to Los Angeles and opens a nightclub to live what Ellis calls a "sort of Playboy-esque, Hefner-esque lifestyle.")
"Most people who meet Lucifer feel compelled to tell him their deepest darkest desires, most women who meet him want to bed him immediately," Ellis explains.
RELATED VIDEO: Tom Ellis Pitches the Best Show on Television: Cockney Sopranos
Meanwhile, as it turns out, playing a smoldering incarnation of the devil has been particularly interesting for Ellis as he himself grew up in a religious household.
"I come from a religious family – my father is a pastor, my uncle, my sister and her husband are a pastor team," Ellis says.
As for how the role bodes with his family, the actor says they understand the lighthearted approach the series aims to take.
"They were really happy for me to get this job and, having seen it, they understand completely what this show is: It's a piece of entertainment where we use a well-known character to tell the story in a sort of fun way, really," Ellis explains.
"It's harmless in that respect. If there's any message at the center of this show it's that maybe people should take a look at their own actions and take responsibility for them," he adds.
And at the end of the day, Ellis slyly admits "it's always fun to play bad characters."
"You know, a lot of actors would tell you that they'd rather play the villain than the hero," he says. "When you're the character, there are no repercussions. So there is a kind of liberating feeling about saying certain things to certain people – and I think that it's always quite satisfying to do that."
Catch Ellis getting up to no good on Lucifer Mondays (9 p.m. ET) on Fox.