"He feels he has let himself down and is very disappointed with the fact that he allowed it to happen," a source tells the U.K.'s Sunday Times.
The paper also says that it's unlikely Harry will attend the opening ceremony of the Paralympic Games this week as originally planned.
Meanwhile, Rosa Monckton, a close friend of Harry's late mother, Diana, Princess of Wales, stands by the 27-year-old.
"He is absolutely entitled to a private life," she tells PEOPLE. "He has a full-time job and he is a full-time serving member of the armed forces. They all need to let their hair down."
Although Monckton acknowledges that Harry is "guilty of a lack of judgment," she goes on to say, "But that's young men for you … It's classic of a young officer. They work very hard and they play very hard. It's a very simple thing."
With reporting by SIMON PERRY