Though Brad Pitt and Angelina Jolie have witnessed poverty in Haiti and visited AIDS clinics in Ethiopia, seeing the devastation remaining a year after Hurricane Katrina hit them hard. "I'm appalled and embarrassed that people don't even have the opportunity to return," Pitt tells PEOPLE. "I do not understand why it's so difficult to get in there and start building. These are our people. We can't be out there selling democracy to the world if we can't even take care of our own here."

So on Aug. 30 and 31, Pitt, 42, made his second trip this summer to New Orleans—this time accompanied by Jolie, 31, and their three children—and announced the winners of a contest he co-sponsored to design housing for the city's ravaged Ninth Ward. Architects Andrew Kotchen and Matthew Berman of the N.Y.C. firm Workshop/APD beat out 3,000 other applicants with their innovative blueprints for 20 affordable, eco-friendly homes and a community center. "If we were to build 50,000 homes to the standards this contest demanded, you could save upwards of $50 million [on energy costs annually]," says Pitt, a self-described "architectural junkie" who put up $200,000 to sponsor the competition with environmental group Global Green USA. Now the group is seeking financing to begin construction in January. Both Pitt and Jolie hope the project will be part of New Orleans' recovery. "It is a wonderful city, and I hope that more families visit and spend time [here]," Jolie says. "[Tourism] is another way we can support New Orleans."

The trip, though, wasn't all business. Pitt and Jolie found a respite in the Big Easy, escaping the paparazzi. They even took son Maddox, 5, to dinner at Galatoire's, one of the French Quarter's oldest restaurants. "It's been great. We've got the whole family here. We've been able to go out and enjoy New Orleans," says Pitt. He laughs off tabloid reports that his romance with Jolie is on the rocks: "It's great," he says. (As for another rumor, Jolie is not pregnant again, says a rep.) In fact, when he finishes shooting Ocean's Thirteen, Pitt and Jolie will head to India to make the movie A Mighty Heart; Jolie is starring, Pitt is producing. Having his family close by certainly suits him, although, three months after the birth of daughter Shiloh, he jokes about life with a newborn. "I guess you give up on sleep. You're not going to get it, so don't even make it an issue," Pitt says, laughing. "The truth is, the kids are so much fun."

They also encourage his socially active side. He'll return to New Orleans in three months to check on the area's progress and film the drama The Curious Case of Benjamin Button. "Being a father means being concerned about the plight of all children, especially the children who don't have the opportunities mine have," Pitt says. "It all concerns me, and the answer is, we could be doing better. It makes me want to get involved."