From left: Jake Harris, Captain Phil Harris and Josh Harris
Are Josh and Jake Harris
at their breaking point on Deadliest Catch
All season long, the ghost of their late father, Captain Phil Harris
, has haunted the crew of the Cornelia Marie
. Relief captain Derrick Ray, who is helming the boat this season, has been trying to chart his own course. But when low crab numbers caused morale to sink, temperatures rise on Tuesday's episode.
"A lot of the other boats struggled, too," Jake Harris tells PEOPLE, "but we didn't really pan out with what we were trying to do, so it was financially, physically and emotionally a really hard season on Josh and me."
In addition to disappointing fishing, "just showing up back to the boat for the first time without the old man and [having] somebody like Derrick running the boat," was rough on the two brothers, says Jake.
Although the younger Harris, 25 – who entered rehab for prescription pain medication addiction in 2010 – tested clean for drugs before, during and at the end of the fishing cycle, Ray loudly voiced his suspicions that Jake was on something during the season.
On Tuesday's episode Ray aggressive decision to call to Dutch Harbor police on Jake is captured.
"I am not a mean-spirited person," Jake says now. "I don't go out of my way to screw the next guy ... [Ray] was just a guy that would go out of his way to make another guy's life harder. I'll never understand that. I don't know what he had going on. Hopefully he was just doing that for the camera and it was not his natural personality coming out."
Josh, 28, adds, "Over the course of time, we're learning to properly deal with situations within reason. Right now, it's having a captain of a boat who has us as deckhands, and we're still part owners, and the captain will come on with a bit of an attitude toward us. They work for us, you'd think they'd be nice as we're the owners but in reality just want to push us down."
To see what happens, watch Deadliest Catch
Tuesday night on Discovery at 9 pm ET/PT.