"It's Kim who is taking it the hardest," a source tells PEOPLE. "Monty knew what was happening. He was prepared."
Brinson's funeral was held in Los Angeles on Thursday. "The wake was very sad and emotional," the source shares, adding, "everyone is just trying to be there for [Kim] and hope that she stays safe."
Richards' niece Paris Hilton was also in attendance.
"Really missing this smile today I love you," Brinson's daughter Brooke Wiederhorn shared on Instagram the day of the funeral.
Those close to Richards, 51, continue to worry about her health and sobriety during this difficult time.
"She is sober right now which is the best thing she can," the source explains. "If [Kim] didn't have her kids, everyone is worried what would happen. She is incredibly fragile right now, and with her history, it's a scary place for her to be."
The former Real Housewives of Beverly Hills star was married to Brinson from 1985-88. He lived with his former spouse toward the end of his battle with lung cancer.
Friends and family have rallied to help the mother of three, including sister Kyle Richards.
Despite the siblings' up-and-down relationship, "Kyle is supporting Kim throughout all of this," a close family insider explained to PEOPLE. "No matter how strained their relationship, Kim and Kyle are sisters and family is more important than anything. Nothing else matters when a tragedy like this happens."