Keri Russell and Matthew Rhys
02/20/2014 AT 01:00 PM EST
Matthew Rhys figures his fans might believe rumors about him on the Internet. But his own mother?
"She calls up going, 'What's going on? I just read this about you,' " the Welsh actor tells PEOPLE. "I tell her, 'You have to take off your Google alerts.' "
The gossip about Rhys, 39, has been focused lately on the rumored relationship
between him and Keri Russell, his recently separated
costar on The Americans
"That is the best one I love," Rhys says of those rumors, which he has had to deny to his mother. "I've told her a million times that there are a million inaccuracies that she's read, [but she still] calls and goes, 'Is this true?' I tell her, 'No! When are you going to get it?' "
"You have to brush that stuff off," he adds.
On top of the gossip, "The sex scenes don't go down well in my house," Rhys says of his family watching the hit FX series
"I've been told for the second season that I must warn them which episodes … they should expect to look the other way. They never get quite used to it."