"No women has ever come up to me and said that my husband was having an affair with them," Teresa tells PEOPLE exclusively in this week's issue. "It's sad that if there are any women around him people take pictures of him and sell the photos to a magazine. I feel bad for him."
For PEOPLE's full coverage of what's going on with Teresa Giudice and her family these days, visit PEOPLE.com/Teresa
Manny Carabel / Getty
After the repeated media reports, Joe says he has had to decline fans' requests for photos.
"Now if a girl comes up to me and says can we take a picture, I'm like, 'I really can't and I'm sorry,' " he tells PEOPLE exclusively.
RELATED VIDEO: Why Teresa Giudice Says She Never Blamed Joe for Her Prison Sentence
Joe also adds that he has a newfound appreciation for his wife – and their marriage.
"Our marriage has always been strong. We never had a problem with our marriage," Joe, 43, says, "But I definitely learned to appreciate her more not having her around for a year."
For her part, Teresa, whose memoir Turning The Tables: From Housewife to Inmate and Back Again, written with PEOPLE Staff Writer K.C. Baker, says her husband has an implicit trust that when he goes to prison, she'll be faithful.
For much more from Teresa Giudice, plus an exclusive interview with her husband Joe as he prepares to serve his own prison sentence, pick up this week's issue of PEOPLE Magazine, on newsstands Friday
"I have single friends so if I'm going to be out and if there's people around us and if there's men around us, I'm not going to be like, 'Oh my God, stay away from us, I'm afraid somebody is going to take a picture of us.' I'm just going to live my life," says Teresa, 43. "Joe trusts me, he knows I'm faithful, I know he's faithful and that's all that matters."
"We've been married 16 years. Listen, we're both devoted to each other, and I don't have a collar around his neck," she adds. "If we didn't want to be around each other we would both go our separate ways."