A former Late Show with David Letterman
staffer tells PEOPLE that the talk show host had a sexual relationship with one of the show's employees in the late 1990s.
The source says the relationship occurred before Stephanie Birkitt, 34, the woman at the center of the Letterman scandal
, worked on the show and involved a different female staffer.
"I'm not surprised this came out," the former staffer tells PEOPLE of Letterman's admission
Thursday night that he had "had sex with women who work for me on this show."
"Even the interns knew stuff like that was going on," says the source.
The former staffer says that Letterman, 62, carried on a sexual relationship with one of his "peers, a woman close to him, not an intern."
At the time the relationship didn't raise too many eyebrows, adds the source. "It wasn't a big deal because he wasn't married. And we heard he had a girlfriend (Regina Lasko, whom Letterman married in March 2009), but she never came around, so it just wasn't a big deal."
"He wasn't considered a sex symbol or anything. In fact, off the air, he came across a lot older than he was."
Various former colleagues say Letterman could be both aloof and charming – but was always extremely private.
"David is a perfectionist, but his cameramen and other staffers respect him and are loyal to him," says one source. "People on his staff genuinely like him."
Another close source, who has a long history with the Late Show
, says that the program's grueling schedule – 15-hour workdays are typical – helped breed inter-office hookups.
"In politics it's the same thing," says the source. "People who live it, eat it, breathe it ... [It's] not some sort of predatory, 'Let's hire beautiful women so we can feast on them’ kind of thing. That's just not the way it works."
The former staffer adds of Letterman, "He never acted inappropriate with any of the female staff. He was very kind, and I never recall anyone saying anything at all bad about him."
With reporting by Mark Dagostino