Joel Madden: I Did 'Right' By Hilary Duff
"I kind of ignored it and tried to just continue to live my life," Madden, 28, tells PEOPLE. "People can say or think whatever they want ...so in my reality, it's kind of irrelevant. I'm always the kind of person that does the right thing and keeps my side of the street clean."
He also confirms what Duff, 19, recently told PEOPLE about how she and Madden keep in touch and have a "great relationship."
"Every girl I've ever dated I have respect for, and I always try to leave them in good terms," says Madden. "I don't see the necessity to be friends with anyone you've dated but definitely out of respect of the time, you have to be kind of respectful and that's what I try to do."
Since then, he's been dating Nicole Richie, 25.
"I don't really talk about her at all but I will say I am really happy and everything's really good for me right now," Madden says.
It's not just his new relationship that's got him smiling. With a new album, Good Morning Revival, coming out March 27, Madden has been having fun traveling with his brother and fellow band member, Benji.
"He's my best friend, so I love spending time with him," says Madden. "It's [also nice] having someone on your team, because the world has a very different perception of you than the person you really are."
In that regard, he says, "I'm just a really normal, sensitive kind of go-about-my business everyday kinda guy. People see the tattoos, and they either read things or they see things and they don't really know that I'm just this guy that gets up and makes coffee in the morning and hangs out with his friends and walks his dog and reads his Bible and goes about his day. I'm not perfect obviously, but I'm just a normal – like any other dude that's 28 and just living his life."
This life includes something else that's perfectly normal, he says: messing up.
"My goal in life is to do the right thing and learn from my mistakes. I feel so bad for someone like Britney Spears who's not allowed to make normal mistakes we all make," he says. "I've had plenty of times on tour or somewhere where I freaked out and went crazy and did things when I was right around her age."
As he sees it, "Mistakes aren't bad, and I've made plenty of them – but I'm always trying to be a better person."