The country star and his wife, Caroline, have been parental figures for the three teenagers after their dad – Bryan's brother-in-law Ben Lee Cheshire – died last November from a possible heart attack. Their mother, Bryan's sister Kelly, died suddenly in 2007 at age 39.
"You don't want to sound like you love having them so much that you're glad it's the situation, but we're honored to be doing what we feel was the right thing," Bryan, who also has two young sons, tells Billboard in its latest issue.
The "Kick the Dust Up" singer has kept an upbeat attitude through tragedy – and his songs reflect it. With a new fan app out now and an album on the way next month, Bryan is aware that he's often been labeled a "frat-boy singer" – but as he prepares to turn 40 next year he's aware that label won't work forever.
"I always say, if I ever get to 25 No. 1s, I'd try to do all 25. I wouldn't want one person not to hear their favorite … At some point, me singing about frat party themes is just not going to be realistic," says Bryan, who's well-known for jokingly shaking it at his shows. "But if I look like a weird old dude up onstage, I'll be the first one to come to that realization."
Another thing he's contemplating deeply? The meaning attached to the Confederate flag, which has come under fire since the Charleston shootings last month.
Raised in the heart of South Georgia, Bryan sees both sides of a sensitive issue: "Where I grew up, I never understood the Confederate flag to be a negative thing."
"But if the Ku Klux Klan is going to walk around and turn the Confederate flag into their deal," he says before pausing. "It's become a symbol of racism to a majority of people. And we live in a country where we have to listen to people's opinions and work it out."
Don't miss a beat of country music news, photos and videos! Click here to get all this and more in the PEOPLE Country Newsletter.