Robertson was suspended by A&E Networks after his comments to GQ magazine ignited a firestorm, but his suspension was quickly lifted, and the network announced plans to start shooting the next season of the show in the spring.
"The family is happy that they can put this behind them," a source tells PEOPLE.
"It got a little out of hand," the source admits. "Phil has never been disrespectful to anyone and treats everyone with kindness. He made some comments off the cuff that he shouldn't have made, and he knows that."
The source says that the headlines didn't cause any drama among the family, who all support their dad.
"The family knows his heart and knows that he wasn't trying to be hurtful," says the source. "They just want to finish celebrating the holidays and are looking forward to getting back to work."
But criticism continued to pour in after the network's decision was announced.
"Phil Robertson should look African-American and gay people in the eyes and hear about the hurtful impact of praising Jim Crow laws and comparing gay people to terrorists," the group GLAAD said in a statement. "If dialogue with Phil is not part of next steps then A&E has chosen profits over African-American and gay people – especially its employees and viewers."