"As much as you want to try and explain your position to people, sometimes at the end of the day it is not worth it," the former Full House star tells PEOPLE. "I need to invest my time in my family and not trying to please everybody or getting them to see my own point of view."
As a contestant on this season of Dancing with the Stars, Bure, 38, has garnered praise and criticism for her conservative beliefs and her "boundaries" that have left her modestly clothed and with fully-dressed partners (even when pro dancer Mark Ballas wanted to show off his abs).
"They didn't show it on the show but the only reason was because I thought it might be a little uncomfortable for my children," Bure says of asking Ballas, 27, to keep his shirt on. "And funny enough, another one of the dancers told me his daughter texted him the day after he went shirtless for the first time and she was not happy."
Aside from her restrictions on DWTS, Bure also faced feminist criticism for a passage in her new book, Balancing It All: My Story of Juggling Priorities and Purpose, in which she states she has a "submissive" role in her 17-year marriage to former pro hockey player Valeri Bure.
"I've had to stop reading my own Facebook page because there can be so much negativity," says the star, who insists she has a very strong voice in her marriage. "But in the past year, I've been working on understanding that not everybody is going to like me or agree with my point of view and it is okay."
For more from PEOPLE's exclusive interview with Cameron Bure, pick up the latest issue of the magazine, on newsstands Friday.