Think of a book club, and the image that comes to mind is one of a group of middle-aged women in a suburban living room, munching on crudités and sipping white wine, talking about The Kite Runner for 20 minutes and then sliding effortlessly into gossip about the markers of suburban ennui: children, husbands, lovers (always other people’s, of course), school boards, nosy neighbors, nosier bosses, and how Linda has lost so much weight since the divorce, maybe we should say something?

My mother has been in such a book club for over 20 years. It meets on the first Monday of every month, and twice a year each member brings in a list of books for the following six months, and then all the women vote. (Paperbacks only, please!) I personally have been in at least four failed book clubs, so the thought of being in one for 20 years seems almost quixotic. Most recently, a co-worker and I decided on a New York-themed book club; we made it through some John Cheever short stories, The Age of Innocence and Washington Square before giving up.

But the book club that met the other evening at the Upper East Side apartment of Susan and Charles Avery Fisher—who is better known as Chip and is the son of Avery Fisher, for whom the hall in Lincoln Center is named—did not seem like the sort of book club that gives up easily. Mr. Fisher, who is 52, runs a company that manufactures a “cranial stimulator,” which delivers an electrical current to the brains of patients suffering from depression; he has also owned a catering company, a cookware store and an Upper East Side ice cream shop called Mr. Chips.

Mr. Fisher started his book club three years ago; it meets only four times a year, always on a Monday evening, in the vast living room of his apartment at Fifth Avenue and 87th Street. (It is the kind of living room where one hardly notices the grand piano in the corner.) Only nonfiction books are read. “I really don’t like fiction,” Mr. Fisher said. “It’s just not my style. I read it occasionally, but it doesn’t really interest me.”

Mr. Fisher often gets the books’ authors to pay a visit to the book club to discuss their books, and usually he invites them back as members. Michael Gross joined after the club read 740 Park, as did Karen Abbott after the club read her book Sin in the Second City, about sisters who ran a Chicago bordello in the early 1900s. “Most authors have been flattered,” Mr. Fisher said. “They rather like the chance to hear what people in a small book club say.” Gay and Nan Talese are on Mr. Fisher’s e-mail list because they are personal friends, though they do not usually attend.

“We have a no-bullshit rule,” Mr. Fisher told The Observer. “You can come if you haven’t read the book, but you can’t bullshit.” Mr. Fisher is on the library committee at the University Club, where he likes to play squash and backgammon. At the meeting the other evening was a new member, Peter Otto, who is one of Mr. Fisher’s backgammon and squash sparring partners. Before the others arrived, Mr. Otto and Mr. Fisher discussed the pro-am (professional-amateur) tournament taking place at the club. Squash doubles, they told me, is quite challenging.

The book under discussion that night was Einstein: His Life and Universe, by former Time managing editor (and current columnist) Walter Isaacson. Mr. Isaacson was, sadly, out of the country, although Mr. Fisher said he had kindly responded to e-mails, and there had been a brief, though ultimately unfruitful, discussion of doing some sort of book club conference call with Mr. Isaacson.

Mr. Fisher’s book club follows a rather set schedule. Members are welcome at the Fishers’ beginning at 7 o’clock, when they may have a cocktail or a glass of wine. (Jackets and bags go in the library.) By 7:30 or so, dinner—made by the Fishers’ housekeeper—is served, buffet-style, on a long table in the dining room, and then eaten on laps in the living room. The other night, there was a tasty curried chicken, macaroni and salad, and two tarts for dessert. When the grandfather clock in the corner chimes 8, it is time for the discussion to begin.

“I’m not a control freak,” Mr. Fisher said, “but I have a routine that works. It’s pleasing for me and it’s not annoying to anyone. Most book clubs meet 10 to 12 times a year. I think that’s a punishing schedule.”

The members in attendance that evening were an Upper East Side hodgepodge; they included Georgia Shreve, the poet and writer who sold her duplex penthouse in Mr. Fisher’s building for a reported $46 million in December; Mr. Gross’s wife, Barbara Hodes, who designs knitwear (Mr. Gross was attending the PEN Awards gala at the Museum of Natural History that evening); an arts and fashion writer named Marcia Sherrill; handbag designer-turned-real estate agent Carey Adina Karmel; art appraiser Catchia Goggin; and lawyer Blake Hornick, who went to overnight camp with Mr. Fisher.

“We’re very liberal about who comes,” Mr. Fisher said. “It’s usually friends of friends. We only had one guy who got kicked out. He was a lawyer we knew. Basically, the first meeting he came to, he had a list of comments about the book. It was like preparing a brief for a litigation trial. I sort of didn’t comment on it, but he got the idea that it wasn’t a great idea.

“This guy’s rather rigid,” Mr. Fisher continued. “He’d offered to give the book club during the summer meeting, when we don’t have air conditioning.” (It would involve extensive renovations to install central air, Mrs. Fisher explained, and they are not allowed to have window air conditioners facing Fifth Avenue.) “Then I didn’t call him with the exact number of people who were going to come. He decided that was a lack of detail he couldn’t live with, and he canceled the book group on the Friday before our Monday meeting. I didn’t think it was like lining up at the door at Marquee with everybody’s name and saying, ‘I know Buffy’ or whatever code name. So we kicked him out. We just said, that’s unacceptable behavior.”

Last Monday, the conversation about Einstein quickly took a turn for the eschatological. “Fate is not the same as predeterminism,” said one member.

“Einstein was not a Calvinist,” said another.

“Oh, no, no, he was a fatalist!”

“I feel like the universe is much more wobbly now,” said another.

“It was a very revolutionary time in human thought,” chimed in another member.

The maid came out of the kitchen and cleared away some plates. Mr. Fisher brought out two framed photographs of Einstein with Susan Fisher’s father, the Princeton astrophysicist Thomas Stix. “Einstein absolutely was just a man on the street,” Mrs. Fisher said.

Someone mentioned that Einstein had an exceptionally large brain.

“I think he had narcissistic personality disorder,” said one woman.

“I hate it when people call it a disorder!” another responded. “All narcissists aren’t sick.”

“We need to develop a narcissistic component in ourselves,” said another woman.

The first woman responded, “I think there’s something to be said for that.”

At 9:30, it was time to select the next meeting’s book. The assembled members finally decided, after a short debate and several rounds of voting, on Dancing Into Battle: A Social History of Waterloo, by the British author Nick Foulkes. Among those voted down: Taxi!: A Social History of the New York City Cab Driver; The Fortune Cookie Chronicles; The Nine: Inside the Secret World of the Supreme Court; Ike: An American Hero; Leviathan: The History of Whaling in North America; This Republic of Suffering: Death and the American Civil War; and Rome and Jerusalem: The Clash of Ancient Civilizations.

“I need someone to be dinner chair for next year,” Mr. Fisher said after the voting had concluded. “I have become increasingly busy.”

“This is like the papacy,” said Mr. Hornick, the lawyer. “You can’t give it up until you die.”

“Book club is my responsibility,” Mr. Fisher said. “Feeding the hordes is not my responsibility.”

“I will bring dessert wine,” Mr. Hornick declared.

“No, don’t bring fucking dessert wine,” Mr. Fisher said. “It’s going to be 80 degrees.”