"I was a terrorized kid," James, 41, said about beatings he received from his father. "I was petrified of my dad … It wasn't so much getting the sh-- beat out of me or getting my arm broken or getting kicked or whatever or punched, it was the in-between time. It was the fear of that happening again."
As a result of his lack of self-worth, he said, "I grew up with a huge amount of shame and fear and abandonment on my shoulders from a very young age, and I think, you know, the way my mind rationalized [cheating], 'Well, you know, I might as well do whatever I can to like run her off, 'cause she is going to find out what I am anyway and leave me anyway.' "
James added, "The struggle within myself for the things that I did ... to damage marriage and my life and everything else, it's all me. I'm doing it because, you know, I've basically never felt good enough for anyone."
James said it was during his rehab treatment that the memories returned. "He beat my ass pretty good a bunch of times," James said of his father, Larry James, who raised Jesse after his parents divorced. "I just remember, like, clinched teeth, strained-neck look on his face. My whole childhood, I never had a chance to be a kid … I was always scared."
Father Denies ClaimOne time, James said, his father chased him in the dark. "I tripped over this little low fence wire and snapped my wrist and I remember my dad laughed at me when I hit the ground and called me a dummy," James said.
In an email to ABC News, Larry James says his son's allegations of abuse are not true.
And while Jesse "knew it was horrible" to cheat on his wife, it was the fact that his life was so good that he says caused him to trip himself up.
"I knew I would get caught eventually, and I think I wanted to get caught. I don't think, I know for a fact now what it was. It was me trying to self-sabotage my life," he said. "I think I do a lot of things in my life that I shouldn't be doing that aren't conducive to being a perfect husband, you know, and the affairs are just one of them."