"I'm not too big on cats and sometimes I'll say something like that and people get so mad at me," the Voice mentor, 37, tells PEOPLE. "But the truth is, I don't care if they get mad."
So what exactly does he have against his wife's furry friends?
"They're sneaky," explains Shelton, who's currently starring in Pizza Hut's barbecue pizza commercials. Though "to be fair, if they could send a Tweet, I'm sure they would say something bad about me."
So while the singer's idea of a perfect day with his wife of three years doesn't include any felines, he does enjoy spending time with her at home (just seriously, keep those cats away).
"Our ideal day is actually nothing," he says. "She's really gotten into horseback riding. She's into English-style riding now. I still love farming and gardening and things like that in the summertime. We have fun when we're together at home, you know?"
For more on Blake Shelton's life at home with Miranda Lambert, check out the latest issue of PEOPLE, on newsstands Friday.