Handler, who grew up the youngest of six siblings, lost her brother Chet in 1984 when she was just 9 years old. The eldest Handler sibling died after he fell off a cliff in Jackson Hole, Wyoming.
"Seeing your parents fall apart is really rough," Handler, 41, told PEOPLE and Entertainment Weekly Editorial Director Jess Cagle. "I wouldn't wish it on anybody."
"As hard as it was for me or for my brothers or sisters – [how did it feel] for my parents to have your own child in danger in that way? And then you can't protect them and you couldn't save them?"
"In hindsight it really kept us as a tight-knit group, because it was so tragic and awful," she says. "Ultimately, it was kind of a beautiful gift because we all value each other so much."
Brian Bowen Smith
Another gift Handler feels blessed with is her memories of growing up the baby of her family.
"They took care of me," she says of her five older siblings. "They took me everywhere, and I was a little rascal. They'd go to their high school parties and bring me because I had this attitude. I was their little entertainment wind-up doll."
"I was, like, 7 or 8, and he would come home and I would make him cereal when he would come home late from school, from college, and I would put it in a bowl and act like I made him dinner, and look after him."
She continues, "He would always come upstairs to my mom's bedroom and be like, 'Where is she? Where is she?' I would be under the bed and he'd grab me and he'd drag me down the stairs while I was kicking and screaming. We played that game all the time."
Handler says that sometimes the memories are enough to sustain her: "You have all these beautiful memories, and that's enough sometimes."
Watch the full Jess Cagle interview below:
Handler's Netflix show Chelsea begins streaming May 11.