"We were in St. Thomas on holiday," when The Daily Show correspondent, 33, decided to pop the question to his girlfriend of two years, he told PEOPLE Saturday at Comedy Central's "Night Of Too Many Stars: An Overbooked Concert For Autism Education."
The couple, who met in 2008 at the Republican National Convention – Oliver was covering, Norley was campaigning – were friends first.
"I finally made a move," Norley, an Iraq War vet, laughingly admits. "I got seduced," adds Oliver.
Life has changed since her time on active duty, and Norley is now enjoying a slower pace of life.
"I was a medic in the Army," she explains. "It was difficult, but right now it's nice that I get to spend time with the love of my life."
"It's the most emasculating thing I could possibly do to go out with someone who has actually done something valuable with their life," Oliver adds jokingly.
"He's got a forum," Norley says on a more serious note. "We still have troops on the ground. No matter whether there are two or 200,000, we have to make sure our hearts and minds are with what they're doing."
The couple plan to wed "some time next year."