In a clip obtained by the New York Post's Page Six (see video player below), West, 38, reportedly yells, "Don't f--- with me" after finding out that producers removed part of his set for his musical performance on Saturday's episode of the NBC variety show. (A source told PEOPLE earlier this week West was "upset as they dismantled half his staging 30 minutes prior to showtime.")
However, a source close to West tells PEOPLE he was not yelling at SNL staff, explaining: "This audio was secretly recorded while he was venting his frustration in a private moment with his team. He found out his stage design was changed and taken apart under the direction of the show's lighting director without anyone's approval."
Warning: Explicit Language
The source continues: "He had spent an entire day rehearsing and [put] a lot of hard work into the performance. Dramatic set changes were made 30 minutes before going live. It should be understandable why he was upset after being completely blindsided."
According to the insider, West was seen hugging SNL creator and executive producer Lorne Michaels after the show. He stayed around to chat with the staff and cast, plus thanked the producers on his way out the door, the source says.
In the beginning of the leaked audio clip, West is heard saying, "Look at that s--- – they took my f---ing stage off of SNL without asking me."
He goes on to mention Taylor Swift following their latest flare-up. "Now I'm bummed. That and Taylor Swift, fake ass." (He takes credit for her fame in his new song "Famous," rapping, "I feel like me and Taylor might still have sex / Why, I made that bitch famous." On Monday, she struck back in her fiery Grammys acceptance speech.)
Then it appears West considers bowing out of his performance, saying, "Now I ain't gonna do this. We're breaking the mother f---ing internet. I went through six years of this f---ing s---."
He continues, "Let's get to it, bro," and repeats himself while escalating the volume of his speech. "Let's get to it, bro!"
After what sounds like a jump in the clip, West compares himself to filmmaker Stanley Kubrick, Pablo Picasso, Paul the Apostle and drug lord Pablo Escobar – who was speculated to be the inspiration behind the title of West's new album, The Life of Pablo.
"Are they f---ing crazy?" says West. "Whoa, by 50 percent. Stanley Kubrick, Picasso, Apostle Paul, f---ing Picasso and Escobar. By 50 percent more influential than any other human being."
That's when he begins shouting again, "Don't f--- with me. Don't f--- with me. Don't f--- with me."
The audio cuts out after this line: "By 50 percent dead or alive, by 50 percent for the next 1,000 years. Stanley Kubrick. 'Ye."
Despite West's SNL debacle, $53 million debt claims, Twitter drama and differences with Swift, a source close to the musician tells PEOPLE concern for his mental health has been blown out of proportion.
"I haven't felt personally concerned about his well being," says West's longtime friend. "Every time I talk with him – which is almost every day – he's extremely good-natured, [and] in a great mood ..."
Reps for West and NBC did not immediately respond to requests for comment about the SNL incident.
With reporting by JANINE RUBENSTEIN