11/17/2012 AT 05:30 PM EST
In the background of every episode of Survivor
, there was Artis Silvester – growling, frowning, pacing. But the 53-year-old computer engineer tells PEOPLE he was having the time of his life, although he still carries a grudge against one player.
You tried out for Survivor several times. Why did it take so long?
I started trying out at season 2. Then I got cancer. They pulled a big mass out of my neck and I lost about 60 lbs. during chemo and radiation. But I kept trying, and I was thrilled to be cast.
Watching the show, you always looked angry. Didn't you enjoy your Survivor experience?
I wasn't angry. It's hard to watch the show now, knowing how much fun I had, and seeing that it was not being shown. I had a blast out there. [Laughs] I need to work on that. When I'm not smiling, I look like I'm about to take somebody's head off. People were scared of me. But the only person I was angry at was Mike Skupin.
Okay, about that. What happened between the two of you?
Without going into detail, let's just say I don't have good feelings about Mike. When you don't care about people as human beings, that has nothing to do with the game. Mike showed me he didn't care. The fact that everybody loves Mike just blows me away.
But what exactly did he do? You complained about him eating the raw rice and diving with the mask, but you did those things, too.
It was more than the rice or mask. Mike does nothing that doesn't serve Mike. When you strike a nerve with me, I don't care about consequences. But maybe my mistake is holding a grudge.
After the diving challenge in episode 3, RC took you aside and made you swim for her, to prove you could. Did that make you angry?
Yes. There were stereotypes out there. They thought I was unable to swim because I'm a black man. And I said, 'I told you I could swim.' I told Mike that, and he challenged me. He said 'Where do they have 20-foot pools in DC?' They made assumptions about me.
You were aligned with Pete and Abi…
No. I was aligned with Pete. Abi was just unfortunate baggage that came with Pete. ... Yeah. She's not who I would choose to align with.
So who is playing a good game out there?
Denise. She has been to every tribal council and has never been in danger. That tells me that she's playing the game well.
What about Lisa Whelchel? How did she do?
I like Lisa. She did things that she felt she needed to do to advance, so I have no ill will towards her. I was a little disappointed in how she did some things, but we had some really great moments together. She was having a hard time, and if she was faking it as strategy, I fell for it hook, line and sinker. We're good.
Didn't you recognize her?
When that show was on, I was too busy trying to get home without getting beaten up. I didn't have time to watch a show where everyone lived a life that I wasn't living.
So you're out of the game. Does that sting?
It's frustrating, but it is what it is. At some point in time, almost everybody goes home. That's Survivor
So what do you take away from the experience?
Even though I'm not happy with my edit, I had an incredible time. I can't thank CBS enough for that. I got to live out a dream. When I was walking away after Jeff Probst snuffed my torch, I thought, 'There are millions of people who would kill to be voted off Survivor