Maybe the late Babe Paley was wrong. You can be too slim and too newly rich, or that anyway was the least ugly explanation of why the board of Manhattan's exclusive River House coop rejected Gloria Vanderbilt's $1.1 million bid to buy a two-story co-op.

In affidavits filed with both the New York State Supreme Court and New York City's Commission on Human Rights, Vanderbilt, a sleek 56, charged that the River House directors had acted on the supposition that black entertainer Bobby Short, her frequent escort, was the man she would be bringing home to dinner and domicile. Added Vanderbilt's lawyer, Thomas Andrews: "The seller's attorney asked whether Gloria intended to marry Mr. Short. It is none of their damned business."

Then the directors, stung by the publicity and the taunts of some of their East Side neighbors, denied that race had played any part in the Vanderbilt decision. Their real objection to Gloria, suggested board president Carl Mueller, was that she is a Seventh Avenue designer now better known for her jeans than for her genes (she is the great-great-granddaughter of Commodore Cornelius Vanderbilt). Though River House has several celebrity owners in residence, including Henry Kissinger and Josh Logan, Vanderbilt's renown was apparently regarded as tacky. "Fame which attends public service and professional achievement," Mueller declared loftily, "is to be distinguished from publicity which is the result of constant cultivation to promote commercial self-interest...I believe that the ceaseless flow of gossip column items about [Vanderbilt's] comings and goings would attract unwelcome publicity to the River House."

The directors also questioned whether, given the "up-and-down nature of the fashion business," Gloria's listed net worth of more than $7 million would be sufficient to back up her offer. "We are convinced that the longer she can drag this out, the more jeans she can sell," declared River House attorney Marion Epley, though he confided privately: "My daughters are furious with me for being against the 'Blue Jean Lady.' "

The financial question, at least, was swiftly laid to rest when Vanderbilt agreed to put $1 million in escrow on the co-op pending the resolution of what could be a lengthy court battle. Meanwhile, she may consider herself a comrade in rejection of Richard Nixon and Diane Keaton, both of whom were reportedly denied entry to River House. There is no danger, however, that the four-times-married heiress and her two at-home children, Carter, 15, and Anderson, 13, will find themselves out on the street. She has three other New York co-ops to her name.

As for Gloria's relationship with club singer Short, things couldn't be better, thank you, or any less likely to end at the altar. Recently the couple co-hosted a party at Maxwell's Plum for the performer's colleagues in the musical Black Broadway. Eubie Blake played piano, and Gloria and Bobby danced the evening away cheek-to-cheek. But a wedding? "I stand behind Gloria," says confirmed bachelor Short, "and I enjoy being with her, but I don't think there's any chance of our getting married. The people at River House have based their objection on a false assumption. That's not the way the world turns."