11/14/2013 AT 10:00 PM EST
His piercing eyes
? His easy smile? Or maybe just his ability to get down and dirty with his two kids in the family's chicken coop?
Whatever it is, Scott Foley is still wondering what earned him the latest title of PEOPLE's Sexy Man of the Week
"I was flattered, of course. My wife
[Marika] was confused. My publicist was surprised. I never considered myself sexy, but I'll take it!" the Scandal
star, 41, jokes to PEOPLE.
The actor has certainly come a long way since his younger years, a time when Foley, then fresh out of high school, found himself entering into "a big geek" phase.
"I was a little bit of a late developer. My face got a little long and weird. I had acne, big glasses and braces," he recalls.
Any evidence of those days is long gone – Foley has since had a variety of roles, from Felicity
's Noel, Scrubs
, The Unit
, Grey's Anatomy
and True Blood
, to his current run as Captain Jake Ballard
. It's his latest turn on television that seems to be in sync with his sexy status.
"Jake is definitely sexier – he takes the cake. Sexiness comes with age and Noel was a little too naïve for there to be anything sexy about him," Foley says. "Jake would definitely win that battle."
While his female fans may agree, Foley jokes that his Scandal costars
, including Kerry Washington, may have a harder time believing his new title. "You'd get nothing but laughter, maybe some pointed disparaging remarks. They're all very kind [so] I think they'll say, 'Of course Scott's sexy!' " he says.
He adds, "[Kerry] would probably laugh and fall out of frame. I think she'd say my sense of humor is sexy about me – I've found her funny bone."
Other than showering several times a day, Foley – who admits to the occasional manicure – keeps his everyday style simple. The one game-changer in his standard routine, however, is shirtless screen time
"I'm a fairly hirsute man, so I'll trim – but not so it's all the same length. That looks weird," he says. "If it's a intensive shirts-off scene, I'll go to a tanning salon one
time. A seven-minute ultra boom tan!"
But even the Sexy Man of the Week doesn't let his good looks get the best of him. When asked what hashtag would best describe him, he quips, "#Foleysanidiot."
For more on Foley, pick up this week's issue of PEOPLE, on newsstands Friday. And check back next week, when we announce this year's Sexiest Man Alive!
Reporting by AILI NAHAS