"I believed Ms. Constand's account," Castor said at a hearing in Norristown, Pennsylvania. "What I think and what is provable in a courtroom are two different things. What I think is that Andrea Constand was inappropriately touched by Mr. Cosby.
"I'm analyzing back in 2005 what I can prove," Castor said. "All of those combined together in my mind created a situation where she had ruined her own credibility and would not be believed by a juror. That does not mean she was not telling the truth."
Castor testified he never interviewed or even met Constand himself. He testified for more than six hours Tuesday at a pre-trial hearing, and was the only witness to testify.
Constand, 42, is the former director of operations for the Temple University women's basketball team who went to authorities in January 2005 with allegations that, in January 2004, Cosby gave her pills at his house that knocked her out and then sexually assaulted her.
Castor, the Montgomery County district attorney at the time, did not file criminal charges, but after Constand sued Cosby, she and the comedian agreed to a civil settlement in late 2006.
Cosby has denied Constand's allegation, saying the sexual contact was consensual. He has also denied similar allegations from more than 50 other women.
"At that point I concluded it was better for justice to make a determination that Mr. Cosby would never be arrested," he said.
Castor says he called Walter Phillips, Jr., Cosby's defense attorney at the time, and told him of his decision to not to prosecute Cosby – now or ever. However, he said he did not have an agreement with Phillips.
"I have heard the term 'agreement' but everybody has used the word wrong," he said. "I told Mr. Philips that I had decided that because of defects in the case that the case could not be won and that I was going to make a public statement because we were not going to charge Cosby.
"Mr. Phillips never agreed to do anything in exchange for Mr. Cosby not being prosecuted," he said. "What he agreed to was my legal analysis was accurate ... and he said he would communicate that to the lawyers who were representing Mr. Cosby in the pending civil suit."
However, Castor testified that a press release he wrote saying Cosby would not be prosecuted did indeed constitute an "agreement" with Phillips.
He testified that he assumes his this press release is why Cosby admitted to what he did during the deposition, which became public last summer.
"Cosby would have been nuts to say those things if he thought there was any chance he'd be prosecuted," he said.
He testified he reached out to Risa Vetri Ferman, the county's former top prosecutor, last September to tell her about the mutual understanding he had with Phillips after hearing she had reopened the case.
"I wrote her an email explaining the situation from 2005 to tell her to tread carefully here because in my opinion she was exposing the county and herself to civil liability because the decision on whether to go forward on Cosby was put to rest by my decision in 2005. And I went in some detail about that so she would have it on record."