"Rob is still struggling and it makes the family sad," the source tells PEOPLE. "He deals with depression, weight issues and low self-esteem, but that's been going on for a long time."
Rob, 28, began avoiding the public eye after battling depression and gaining over 100 lbs.
"The family, especially Khloé, is very much there for him but it has to be his own journey to figure out," adds the source.
Dimitrios Kambouris / Getty
"He's such a great guy. That's also what's so infuriating," Khloé said. "He's charming, charismatic, everything. But I'm like, where is your will to live? Where is your will to live life? Let's be around people!"
In recent years, Rob has skipped out on family events, including sister Kim Kardashian West's wedding, but appears to be working to get back in shape.
"I've been working on my patience," Khloé told PEOPLE in October. "I've been really trying. But I have my moments and I do snap … I can't win life for Rob. I have to wait until [he's] ready to do it for himself."