IN A WORLD GIVEN OVER TO SELF-analysis, navel-gazing and parent-blaming, pianist Peter Duchin was a failure. "He's been untroubled because he managed to forget," says writer Brooke Hayward, 59, his wife of 10 years. While others endlessly rewind and replay their childhoods in search of insight, "Peter had the opposite desire," Hayward says. "He didn't want to know."

These days, the Manhattan-based Duchin is a—some would say the—society bandleader par excellence, playing more than 100 gigs a year. But he is equally famous for having led a through-the-looking-glass life marked by tragedy, privilege, politics, Jackie Kennedy Onassis, Kim Novak and thousands of parties. At 59, he has finally written a memoir, Ghost of a Chance. "It was just going to be an anecdotal book about things I saw from the bandstand," says Duchin. "But I'd been talking to Jackie O, who was a really close personal friend, and she said, 'Look, there's more to talk about because your life is so weird and there's been so much tragedy and pain. If you have the guts, do it.' "

The tragedy in Duchin's life began early. His mother, prominent New York socialite Marjorie Oelrichs, died of childbirth complications six days after he was born, in 1937. Duchin himself barely survived. One of his lungs collapsed right after he was born, and he spent months in an oxygen tent. Almost immediately, his father—the dashing bandleader Eddy Duchin—turned him over to family friend Averell Harriman, later governor of New York, and his wife, Marie.

Duchin says he was always hurt and puzzled by his father's abandonment but now believes that "he must have blamed me to a certain extent for the death of his wife, whom he loved very much." At the age of 10, he moved in with his father and stepmother Chiquita. That reunion ended when Eddy died of leukemia three years later. Peter eventually went back to the Harrimans. Remarkably, Duchin says, he has no memory of the years with his father. Until he started the book, he didn't seek information about his mother either—though he had plenty of opportunities. "To think that I often stayed in Newport [R.I.] with people who knew my mother and my mother's family and never asked about her—that's really weird," he acknowledges.

Before he started on the book, Duchin says the unexamined life suited him just fine. "I don't know if I was scared or what," he says. "I've never been to shrinks. I never wanted to. It wasn't my style."

His wife—herself the author of the bestselling memoir Haywire, about her own celebrated and unhappy parents, theatrical producer Leland Hay-ward and actress Margaret Sullavan—thinks his forgetfulness was a survival mechanism. Even though facing the past was "a risky process," she thinks that, ultimately, "it will be immensely important to Peter that he did the book, that he had the courage."

At Arden in Woodbury, N.Y., Harriman's 40-bedroom estate on the Hudson River, Duchin learned to hunt, play polo and be a people pleaser. "I tried to be liked and charming so I'd be asked to stay around," says Duchin. That charm came in handy at Yale, where he studied music, and during one memorable weekend when he romanced Kim Novak, who played his mother in the 1956 movie The Eddy Duchin Story. She was 22; he was 17. "Kim is just a wonderful woman," says Duchin, when pressed for details.

Duchin knew the blue-blooded—the Whitneys, the Dukes, the Phippses—and the hot-blooded, most notably Ava Gardner, with whom he had an affair in 1962. "She was a woman who really knew a great deal about all levels of life from the dregs to the top," he says of the actress.

With his first wife, Cheray (the mother of his three children—Jason, 30, an actor; Courtenay, 28, a graphics designer; and Colin, 26, a potter), he was a frequent guest on Aristotle Onassis's yacht. He describes the Ari-Jackie union as "an extremely intense, close relationship," adding, "I daresay she loved him very much at one point."

He was much less impressed by Pamela Digby Churchill Hayward Harriman, Averell Harriman's third wife (and wife Brooke's stepmother during Pamela's 10-year marriage to Leland Hayward). "I think I write about her honestly," says Duchin, who portrays the current U.S. ambassador to France as controlling and conniving: "Pamela's focus became Averell and what was good for her and him." Duchin claims, among other things, that she cheated him out of land that had been promised to him by Harriman. She won't comment on the book.

With Ghost of a Chance out of the way, Duchin is back on the road. Between jobs he retreats to his country house in Connecticut, where he gardens and listens to Bach. In those baroque strains, he hears the echo of personal rhythms. Bach, says Duchin, "is highly structured, yet it's immensely free." As is Duchin. He has faced his music; now he's ready to dance.