10/30/2013 at 02:00 PM EDT
After the Jonas Brothers
abruptly canceled their tour
three weeks ago – and went silent as they contemplated the band's future
– reports surfaced suggesting that Joe Jonas
had a drug problem.
Now the singer, 24, wishes to set the record straight, telling PEOPLE: "It's ridiculous!"
"We're not saying anything for two or three weeks and I'm a drug addict!"
What upset him even more is that his girlfriend, Blanda Eggenschwiler, was also dragged into the rumors.
"It's obviously not true. I've never touched heroin in my life. Neither has my girlfriend," he said. "It's hilarious to me, but also frustrating as well. It pissed me off when it's about my girl and even about myself."
At one point, he said, people even approached him personally about it.
"I was walking down the street and people are asking me if they can give me a hug and how rehab is," says Joe. "I'm like, 'You tell me because I've never been.' "
So why didn't he speak out right away to correct the impression?
After Nick Jonas, 21, first confessed to older brothers Joe and Kevin that he had concerns about the band – they then unanimously decided to call it quits – Joe says: "We all had to deal with it in our own way. I needed time to clear my head."
Already, that is a thing of the past. Looking ahead, Joe says, "I'm genuinely excited."
For more on the Jonas Brothers' decision to call it quits, pick up this week's issue of PEOPLE, on newsstands Friday.