Chatting with some Australian competitors in the crowd at the preliminary contests of the Paralympic-style games this weekend, in Orlando, Florida, Harry said the Queen "liked" their teams' video.
"He said she really wanted to send her own message, but they squashed the idea because then she'd have to do the same for everyone," Dani Moffitt, a former communications specialist in the Australian army, tells PEOPLE.
"I told him it was a shame he couldn't bring her, and he said that she's really busy and that she has about 10 or 15 functions to go to every day. She's a busy lady!" Moffitt said.
Want to keep up with the latest royals coverage? Click here to subscribe to the Royals Newsletter.
Harry, 31, channeled a bit of the Queen's comic spirit himself:
As he sat among the crowd watching sitting volleyball, one of the Australian team members asked Harry if he wanted to take his jacket, Moffitt said. Harry had a teasing retort at the ready, shooting back with, "You need to wear that and hide those little arms of yours."
Taryn Moran, a former medical technician in the Australian army who retired due to non-physical injuries, was with Moffitt at the games and says Harry was "so lovely, so personable."
"He has served himself and knows exactly how we all relate to one another. The Invictus Games are becoming so huge," Moran says. "It reminds us that we are remembered and respected as well. It's a wonderful thing that unites us all no matter what country you come from.
"There's a sense of comradeship."