"This week has been kind of a rough week for us," DiMarco tells PEOPLE via his translator of working with Peta Murgatroyd after last Monday's live show.
In the package before their dance last week, DiMarco was shown calling himself a "front-runner" and commenting that he didn't see any of his fellow stars as competition.
That hit the model hard.
"So much happened in our package this past Monday on the live show, and I feel like it was kind of a negative perspective of me," says DiMarco, 26. "It showed me as kind of being arrogant and cocky, and that's not who I am."
RELATED VIDEO: Nyle DiMarco's DWTS Vlog, Week 6
"It was so hard for me to say anything because I was misperceived. And being deaf and also being a minority, I have a huge responsibility to represent the deaf community well," he adds. "And, on top of that, I also have my own foundation and I'm the face of that foundation and I can't have that image."
RELATED VIDEO: Nyle DiMarco's DWTS Vlog, Week 4
But DiMarco hopes he and Murgatroyd, 29, can bounce back with this week's dance, which will be set to U2's "Beautiful Day."
"[The song is] actually about a man who has lost everything until he finally finds joy, and that correlates with me and finding my joy on Dancing with the Stars," he says. "So we'll see what happens live."
Season 22 of Dancing with the Stars airs Mondays (8 p.m. ET) on ABC; to vote for DiMarco and Murgatroyd, dial 1-800-868-3409.