Samuel Riley Pierce Jr. is the member of the Reagan Administration best known for being unknown—even to the man who appointed him. Pierce, the Secretary of Housing and Urban Development, was meeting with some mayors in the White House last June when the President stopped by. "He shook my hand," recalls Secretary Pierce, "and said, 'Hello, Mr. Mayor.' " Several seconds passed before Reagan realized his mistake and apologized. No problem. "Heck," shrugs Pierce, "I've done that."

Pierce, who has now served under three Republican Presidents, doesn't mind being inconspicuous. His nickname in the capital is "Silent Sam, the Invisible Man." "I think it's more important to get things done than have my picture taken," he says. "I guess it's very different in Washington to keep your mouth shut."

After nine months in office, the taciturn Secretary can let his accomplishments speak for him. As HUD's boss during the longest housing slump since World War II and the only black in an Administration unpopular with blacks, his position is not an enviable one. "There's a feeling in the black community that the Reagan Administration is not a friend of blacks," allows Pierce, "but the President is not saying, 'Forget them; let 'em die.' We're not cutting back that far." Pierce stays in touch with black leaders and has lobbied within the Administration for an extension of the 1965 Voting Rights Act. His quiet efficiency has won grudging admiration from such former critics as Richard Hatcher, the black mayor of Gary, Ind. "I now have a basic respect for him," says Hatcher. "Under the circumstances, he's doing the best he possibly can."

The circumstances are unsettling. "The housing industry is sick," says Pierce, 59, admitting there's little to do about it except "pray interest rates and inflation go down." As spurs to new housing, Pierce has proposed an easing of building codes to cut construction costs and tax incentives for businesses that build in distressed areas of cities. HUD's 1982 budget has been cut by nearly 20 percent, yet Pierce has managed to save the FHA home loan program for now and got a one-year extension for the Urban Development Action Grant designed to stimulate decaying communities. "That was one of the first and few defeats [Budget Director David] Stockman suffered," Stockman aide Ed Dale reports. "The President decided in favor of Pierce."

No one was more surprised than Pierce himself when he was asked to head the 15,500-employee agency. A modest contributor to the Reagan campaign, he first met the candidate at a fund raiser. "At the time," remembers Pierce, "I thought, 'I'll never see him again, except on TV.' " However, when Hispanic business consultant Philip Sanchez turned down the housing post (which often goes to minorities), Pierce was summoned to California. He had been recommended by "Kitchen Cabinet" member Alfred Bloomingdale, a client of Pierce's New York law firm. Chuck Stone, a black editor at the Philadelphia Daily News and longtime acquaintance of Pierce, says the Secretary "has the three B's Republicans like—brilliance, blackness and blandness. Republicans don't like audacious blacks."

The most audacious thing about Pierce is his list of achievements. He grew up on Long Island, the son of a self-made man who parlayed a handyman's job at a posh country club into a cleaning business and eventually a real estate and stock fortune. A competitive youngster, Sam was graduated Phi Beta Kappa from Cornell in the class of 1947. After law school he became a U.S. attorney in New York and then assistant to the Under Secretary of Labor in the Eisenhower Administration. He was appointed to a state judgeship in New York before becoming the first black partner in a prestigious Park Avenue law firm. In 1964 Pierce was elected the first black to sit on the board of a Fortune 500 corporation—U.S. Industries—and in 1970 he was named the first black to serve as general counsel to the Treasury.

Pierce gave up his $300,000-a-year law practice to accept the $69,630 job as HUD Secretary. He now shuttles between New York, where he shares a Central Park West co-op with physician wife Barbara, and Washington, where he sometimes dines with their daughter, Victoria, 32, an official at the Environmental Protection Agency. He traces his ability to cope in the Reagan Cabinet to his distinguished series of "firsts." "I don't feel lonely because I'm the only black," he says. "I've been in that role too many times."