"Over the past 20 years, I've heard people saying, 'Isn't that the dude who hit Halle Berry in the head?' " he tells PEOPLE.
Justice continued his success on the baseball field for years after the divorce, winning two World Series pennants and the 2000 ALCS MVP award. But even his strong play didn't stop fans from heckling him over rumors that he had abused his famous ex-wife. "They'd scream, 'Hey Justice, hit the ball like you hit Halle!' " he remembers.
His current wife Rebecca, whom he married in 2001, had heard the stories too before they started dating, but chose to give him the benefit of the doubt. Unfortunately, Justice says most people will "believe everything they read," like the flight attendant he says came up to his seat recently and asked, "Aren't you the guy who hit Halle Berry?"
After keeping mostly silent about the situation over the past two decades, Justice finally let loose last week, sending a barrage of tweets directed at Berry.
In the messages, some of which appear to have been deleted, Justice, 49, repeatedly stated that he never hit his ex-wife during their relationship. While Berry has spoken publicly about an ex-boyfriend she says caused permanent hearing loss to her left ear, she has never named her abuser.
VIDEO: A Look Back at Halle Berry's Marriage to the Man She Called 'the One,' Olivier Martinez
Sources close to the Oscar winner, 49, responded to the tweets, telling PEOPLE that, "Halle and David haven't spoken for years and when they did last, there was no animosity between them whatsoever. They were really friendly. This is coming out of nowhere."
But for Justice, setting the record straight was a long time coming. "For so many years, there has been this cloud upon the guys who were associated with Halle back then," he says. "Halle has said that she was in abusive relationships but she didn't name who the abuser was, so for many years I know a lot of people thought it was me."
In sending those tweets, he says, "All I was trying to do was let the world know that David Justice is not the person who hit her on her head and caused her hearing loss. Not only did David Justice not do that, David Justice never hit her, period."
A Difficult DivorceJustice acknowledges that Berry has never accused him of domestic violence. But the circumstances of their split, he says, led many people to presume his guilt. When they divorced in 1997, Berry filed for a restraining order against Justice, alleging that he had lingered outside their home for four hours "demanding entry" to retrieve some of his things. At one point, according to her request for the restraining order, Justice was overheard saying that if she didn't let him in, "he would break every one of the f------ windows … and break the door down." Berry was granted a temporary restraining order.
Justice denied the incident, saying through a spokesperson at the time that "I pose no threat to Halle." He had previously accused her of overreacting. "Every time she saw a picture of me with a woman, any woman who might have been standing by my side, she thought I was cheating," he told PEOPLE when they split in 1996. "I've never known a girl who could throw a tantrum like she does." Years later, she would tell Parade magazine that the split from the man she once called her "prince on a white horse" led her to attempt suicide.
Justice knows people will wonder why he didn't speak up sooner, and that some will think he's bringing this up in the midst of Berry's divorce from Olivier Martinez in an attempt to see his name back in the headlines. But the former three-time All-Star and two-time Silver Slugger award winner says, "I've had enough fame for a lifetime," adding that he "didn't say anything before because I didn't want to battle with her in the media."
Justice explains that he avoided fighting with Berry in the press because he didn't have the tools he needed to win. "As a baseball player back in those days, there were no publicists, I didn't have a team. Halle had managers, lawyers, publicists, etc. It was Team Halle versus me." By keeping silent, Justice adds, "I thought I was taking the high road, and as a result a lot of stuff was said about me, and because I didn't saying anything back, a lot of people thought it must've been true."
Justice also believes that his tarnished reputation directly affected his marketability as a player. "I will never be able to quantify it, but I guarantee that I have lost a lot of endorsement deals behind this cloud over my head, no question," he says.
'Protecting My Kids'Today, the father of three – David Jr., 15, Dionisio, 13, and Raquel, 11 – focuses on raising money for his charities and is an active member of his San Diego community, coaching baseball, soccer, basketball and football. "My kids know what their dad is about, but the one thing that I could not have was my kids' little friends and their parents, or social-media-type people, seeing this particular thing attached to me," he says. "I'm in the community trying to coach, being a mentor. I'm not perfect, but don't ever say that I hit a woman."
As for why he decided to speak out now after all this time, Justice says that while he isn't the type to read up on Berry in the press, a recent article on her divorce that made reference to the abuse opened old wounds. "This particular time I happened to click on an article, and the first thing I thought was, 'Wait a minute now, they're saying something that is so far from the truth,' and I wondered how many people are going to read this and believe that it's true."
In addition, his daughter Raquel is an aspiring actress and just landed her first role on ABC's upcoming show Recovery Road. "If she's going to be floating around Hollywood, I don't want anybody bringing this up to her," he says. "All of us who have kids want to protect them as much as possible, and I felt like this is part of protecting my kids against something that is so bad, so horrific, that her dad can't be attached to it."
Now that everything is finally out in the open, Justice insists that he doesn't "hate Halle" and doesn't "care what's going on with her affairs." In fact, the day after his online rant, he posted, "I just want to thank Halle for finally squashing all of the rumors in the past." While Berry never actually released a statement to squash the rumors, Justice says that when sources close to her reaffirmed that she had never publicly accused him, he "took that as confirmation enough."
Still, Justice admits, "I do wish that she would have spoken out. She can keep mum on who the actual person is, but I think she owes it to the other guys to clear their names. That's a very simple, one-line statement." But he also realizes that, "if after 20 years she's not going to say it, she probably never will."
For now, Justice feels satisfied that he's had the chance to say his piece and clear his name. "I've been carrying this for a long time," he says with relief. "I feel like I've been doing someone else's time for too long."