"I said, 'I love you,' and then I felt him just kind of leave, and it was shocking," Richards, 51, told Entertainment Tonight on Tuesday. "I just didn't expect it."
Brinson and Richards, who share 29-year-old daughter Brooke, were married from 1985-88 but remained friends for years. He even moved in with the former Real Housewives of Beverly Hills star while ailing and was her staunchest supporter after she relapsed with her substance abuse and was arrested twice last summer.
"He was glad that he was going first because he didn't think he could bare it if it was me and he had to be left here," Richards told ET of his passing.
VIDEO: Does Kyle Richards Blame RHOBH for Her Roller Coaster Relationship with Kim?
Sources previously told PEOPLE that the former child actress' family and friend worried about her sobriety slipping as she mourns Brinson.
Still, "I'll just say my recovery is going amazing," Richards said.
Richards now appears on The Mother/Daughter Experiment: Celebrity Edition, which airs Tuesday at 10 p.m. ET on Lifetime.