"The women that spoke out before me, gave me the courage to tell my story but it was not easy," Johnson, 62, told PEOPLE in an interview Thursday night.
In the mid-'80s, Johnson was a model and aspiring actress when Cosby invited her to his Manhattan townhouse for an alleged "audition" for The Cosby Show,, she says.
"At the time it happened, Bill Cosby was on top of the world and he was very powerful in Hollywood and I was in Hollywood trying to make this transition from model to actress, and so I had a lot of fear about speaking out," she says.
"One of the things I found out in coming forward is it's the norm not to tell. Often, women don't."
She recalls how he offered her a cappuccino and, after one sip, she felt woozy and immediately knew she had been drugged.
"When I took that drink, it was like night and day for me," recalls Johnson. "I went from rehearsing a scene with Bill Cosby to this man just drugged me. I have no idea what the drugs were, but I felt it in an instant and it was very intense."
Became DisorientedShe remembers becoming disoriented, and recalls, "I knew that it was getting worse and that's why I became more agitated, trying to defend myself. I knew from the moment I took the first sip. It hit me that fast."
After Cosby grabbed her waist, she says, "I confronted him on it and called him a mother-----. I don't think Bill Cosby was used to being called a mother------, and the expression on his face showed that. I knew what he had done and called him on it. I was furious, and he had chosen the wrong person at that point. It was a cowardly act."
After she confronted him, she says, "He was agitated. He was pissed. He grabbed my arm and I didn't know where he was taking me. I was stumbling and I realized we were outside. He was flagging a taxi and opened the door and he rushed me in. When I got in, I said [to myself], 'Did I just call Bill Cosby a mother------ instead of worrying about myself and being drugged?' That's the culture, you blame yourself.
"I didn't know that he was raping women. That came out years after, and … it became more ugly and more sinister and more violent and dangerous. But at that point, I was an aspiring actress and I was scared."
Johnson told no one. "I was so devastated and I was so disappointed, and it was as if a family member had betrayed you," she says. "He was so revered and this was such a powerful guy and he could destroy me."
As more women came forward during the past few weeks, she says, "I thought I'm the lucky one, because I didn't get raped."
As she wrestled with the thought of going public with her recollection, many cautioned her against it. "Most of the people I spoke to were afraid for me to speak out," she says. "They said, 'Don't say anything,' but that's the culture of being a woman, you keep it to yourself."
"This was not easy for me to do, particularly with the racial climate today and what is happening with black men," she says, "but this was bigger than Bill Cosby – this was about sexual violence against women, and I think by telling my story it allowed me to shine a light and reveal that the shame of being sexually abused is often kept in the dark."
Has to Set an ExampleAs a mother to daughter, Anansa Sims, and a grandmother of three, Johnson also felt she had to set an example.
"You have to take action," she says. "Bill Cosby took my power that day, and I took my power back today. Maybe it took 30 years, but I did."
Cosby's attorney, Martin Singer, has denied all previous accusations of sexual misconduct by his client.
"The new, never-before-heard claims from women who have come forward in the past two weeks with unsubstantiated, fantastical stories about things they say occurred 30, 40 or even 50 years ago have escalated past the point of absurdity," Singer said in a statement on Nov. 21. There has been no comment on Johnson's story, which surfaced publicly Thursday, in an essay she wrote for Vanity Fair.com.
Johnson tells PEOPLE that she has no plans to sue Cosby for any type of damages.
"I want nothing. I want no money and I wouldn't touch that money. I came out because I wanted to stand beside these women and give voice to women out there who have been sexually assaulted, to speak up and speak out and speak loudly."