When a visitor presses the apartment buzzer in a Victorian building in London, the master of the household answers personally. "Come right up," crackles the familiar nasal voice over the intercom. And there, on the landing above, awaits the small, hunched, white-haired figure of former Prime Minister Harold Wilson, 11 years into retirement. Now 71 and a peer, Lord Wilson had been enjoying the comfortably private life of an elder statesman until the publication earlier this year of highly controversial allegations by Peter Wright, a former British counterespionage official. According to Wright, there was a CIA-inspired plot within Her Majesty's secret service to bring down Wilson and his left-leaning Labor Party government in the mid-'60s. Wright's sensational autobiography, Spycatcher, has been a runaway best-seller in the U.S. and a clandestine success in Britain, where Prime Minister Margaret Thatcher's success in banning domestic publication of the book on security grounds created a hot market for smuggled copies. Throughout the furor, Harold Wilson has maintained a measured restraint about the book. Until now. "It's all meant to be high drama, 20 years late," he says. "An awful lot of it is rubbish, guesswork, just tripe. I don't think it will be one of the great history books of the period."

Pouring himself a sherry, Wilson settles into a sofa. To begin with, he says, he had never heard of Peter Wright before Spycatcher. In his seven-plus years as Prime Minister, he adds, he had only a few face-to-face meetings with intelligence officials, discreetly referred to as "our friends." He relied on senior civil servants of unquestionable integrity to stay in contact with the chiefs of MI5 (security) and MI6 (secret intelligence).

According to Wright, Wilson came under MI5 scrutiny, largely at the instigation of the CIA's longtime chief of counterintelligence, the late James Jesus Angleton. Angleton cited a source, never identified, who claimed Wilson was a Soviet agent. Wilson had been connected with a timber importing firm and his business contacts in Eastern Europe had raised suspicion. Partly as a courtesy to an allied intelligence agency, MI5 took the extraordinary step, alleges Wright, of opening a secret dossier on Wilson in 1964, when, at the age of 48, he became Britain's youngest Prime Minister in nearly two centuries.

Wilson acknowledges now that he heard speculation about CIA hostility to his government. "But there was no real proof, nothing to justify it," he says, "so I concluded that it was sheer invention. I spent a lot of time in the U.S., and I have a lot of friends there." Yet after Washington had been rocked by repercussions from the Watergate burglary, there were disturbing echoes in London. As Wright tells it, a cabal within MI5 sought to discredit the Labor government, in hopes of returning the Conservatives to power. In 1975, during the last of his four terms as Prime Minister, Wilson seemed to hint at political dirty tricks in a curious statement issued by his office. "Over the period of some years there have been eight burglaries of premises where Mr. Wilson's private papers were kept," it said. At the time, Whitehall was rife with rumors that the break-ins were the work of Britain's secret service.

Wilson does not dismiss Wright's book in toto. "Some of it may be true," he concedes, "but I just don't know how true. I think we are a very long way from knowing all the facts." If an MI5 plot existed at all, he is certain that it involved "a very small number of low-level people" who were "daft and incompetent" and that it did him little or no political damage. Why, then, did he resign abruptly in 1976 with more than three years remaining in his term? "I had been in office long enough and wanted to give other chaps a chance," he insists. "I'd seen too many stay on when they should have gone." Queen Elizabeth herself attended his farewell dinner, poignantly underscoring the rise of an industrial chemist's son to the highest level of international statecraft.

As his mind ranges back over the years, the former Prime Minister reaches for his ever-present pipe. During his years in office, the elaborate lighting-up ritual always provided a smoke screen, allowing Wilson time to polish a witticism or find the right turn of phrase. These days at home he lights up only in his study and his bedroom, not in Lady Wilson's sitting room. "The whole family hates his pipe smoking," says Mary Wilson, his otherwise loyal wife of 47 years. "Little bits of burning stuff fall out of the pipe," she says, pointing out burns in the carpet. "His suits and ties have holes in them too."

Wilson enjoys his cluttered study. "I like it all around me," he says, sweeping his arms around the room and pointing to the bust of Lincoln, the biographies of such distinguished predecessors as Disraeli and Churchill, the paintings presented to him by Soviet leaders Leonid Brezhnev and Aleksei Kosygin, and a prized signed photograph of John F. Kennedy, taken just three weeks before the President's assassination.

Even now Wilson does not accept the notion that he is retired. His appointment book is filled with dinners, meetings and receptions, and he makes almost daily sorties to the House of Lords where, as Baron Wilson of Rievaulx, he sits on the Labor benches. Though he is provided with a government car and chauffeur, he prefers to walk the "17 or 18 minutes" from his Westminster home to Parliament Square. A couple of weekends each month are reserved for family. Son Robin, 44, lectures on math at Oxford and has twin 12-year-old daughters; son Giles, 39, is a schoolteacher near Southhampton. And the elder Wilsons vacation regularly in their cozy, three-room bungalow in the Isles of Scilly. Tradition has it that retired premiers live like pampered country gentlemen, but the Wilsons' style is more thrifty. When they travel by rail, they claim the half-fare for senior citizens.

Toward Wright, 71, a fellow senior citizen who now lives in Australia, Wilson shows characteristic forbearance. Detractors have accused the former MI5 officer of betraying secrets out of pique over a pension he did not receive. "I have never tried to reach conclusions about people who write books," says Wilson. "I write them myself sometimes." He has mixed feelings about the Thatcher government's dogged worldwide campaign to suppress Spycatcher. He defends regulations that place classified documents off-limits for 30 years, yet he is skeptical that any government can succeed in keeping all its secrets bottled up that long. "There are a lot of things governments want to cover up that don't last 30 minutes, let alone 30 years," he observes from experience. "I wouldn't myself have made such a fuss about it."