The Harlem house had been empty for eight years when in 2010 James Fenton, the British poet, and Darryl Pinckney, the African American cultural critic and author, decided to rescue it. They had seen the place mainly by flashlight, because the windows (there are more than 50) were covered in plywood and sheets of plastic. Inside, there were 17 closet toilets and an indeterminate number of one-room apartments. There were dead pigeons and crack vials. The basement was under water. Someone had been systematically breaking in to steal the lead pipes and had opened the water main.

It was, as Mr. Fenton said, totally disgusting and horrible.

But it was also beautiful, at least on the outside. Built in 1890 as a family home for John Dwight, a founder of Arm & Hammer, the house was a fanciful shape, with a stack of four oval rooms, one on each of four floors, that ballooned out onto 123rd Street, and an imposing Neo-Renaissance style entry, with an arch and pillars. More recently, it had been a single-room occupancy building and a place of worship for a religious organization (which had sliced off half an interior oval wall and boarded up the stairs). It had also served as a medical facility and, for one year, in 1937, the home of the Harlem Community Art Center, a Works Progress Administration effort headed by Augusta Savage, the African American sculptor.

Its history was a typical Harlem story, with waves of progress and regression. Dwight family descendants showed Mr. Fenton and Mr. Pinckney a letter a family member had written in the late 1930s that urged the house not be sold to “negroes.”

“With a small ‘n,’” Mr. Pinckney said. “Even when we bought 11 years ago, there was still a lot of anxiety about what newcomers to Harlem meant.” In their first few months, camping in the cold house, their doorway was littered with leaflets protesting gentrification, and its plywood barricades graffitied with slurs about Mr. Fenton’s race and the couple’s sexuality.

In 2006, developers had planned to turn the house into condominiums. By 2008, that was clearly a terrible idea as the housing market collapsed, and the developers walked away. Two years later, Mr. Fenton and Mr. Pinckney bought the place for $1.85 million, and in the last decade, they have poured many more millions into it, as Mr. Fenton said cheerfully, in a passion project that has been, as their architect, Samuel G. White, put it, restoration by excavation.

That’s because behind the weird, haphazardly constructed interior walls of the SRO and the mutilations of the religious group, were delicately carved wooden mantels and fire surrounds made from marble and onyx. One oval room, walls clotted with paint, turned out to be paneled with elaborate mahogany carvings.

Now that their work is mostly done, Mr. Fenton, 72, and Mr. Pinckney, 68, are selling and moving to England. With a price of $8.5 million, it is now the most expensive listing in Harlem. Political fatigue and the pandemic have played a part in their decision, accelerating what had been far-off plans for retirement.

On a recent Sunday, they gave a tour of what has become a wonderland: gleaming wood paneling in the oval rooms, and faux bois and faux porphyry, where the paneling was stripped off; a glittering gold leaf and lapis blue frieze in the front hall; rooms painted in rich, tropical colors and a library bursting with more than 10,000 books in which you will find the complete diaries of Samuel Pepys, among other treasures. That afternoon, on the bottom edge of a linen curtain that was pooling on the rug, sat a fat tabby named Alex.

“Welcome to Jamesland,” Mr. Pinckney said. “It is always an adventure. He can’t help himself. I just sit back and watch things transform.”

In the Beginning

A former war reporter, foreign correspondent, theater critic and librettist, among other exploits, the Oxford-educated Mr. Fenton is also a scholar of architecture and interior design. He can declaim, slowly and with great precision, about the visual idiom of seemingly any architectural period. He knows what a dado is.

He also has a long habit of courting danger, domestic and otherwise. When Saigon fell to the North Vietnamese in 1975, Mr. Fenton, a stringer for The Washington Post and other papers, hitched a ride into the city on one of the conquering tanks. In the 1980s, on an expedition in Borneo with the British adventurer Redmond O’Hanlon, he nearly drowned in a river.

“James is a Renaissance man,” said Martin Amis, a friend since college. (With Christopher Hitchens, the late cultural gadfly, Mr. Amis and Mr. Fenton were once a fierce literary trio, employed in the 1970s by The New Statesman, the progressive British political and culture magazine.)

“I couldn’t be less of an expert on this kind of thing, but he just puts his mind to it,” Mr. Amis said. “Clearly what stimulates him is an absolute challenge.”

For the Harlem mansion is not the first derelict property Mr. Fenton has wrestled into shape. The “Scene of the Crime Farm” is how his friend Ian McEwan described his last place, a run-down establishment on 100 acres outside of Oxford.

“It looked like the kind of location a body might be found in an advanced state of decomposition and I earnestly advised James not to buy it,” Mr. McEwan told The Economist in 2019. “I told him he should look for an old vicarage somewhere.”

Mr. Fenton worked on it for 20 years, creating a showplace of a garden and designing an addition to the farmhouse around an enormous French Renaissance fireplace that Mr. Fenton had found at auction and shipped to England. (In the 1980s, he had worked on an early version of “Les Miserables,” Cameron Mackintosh’s pop opera blockbuster, which landed on Broadway in 1987 and paid him royalties — the ability to fund construction adventures — for a long, long time.)

He and Mr. Pinckney met in 1990 at the Paris Bar in Berlin, introduced, by accident, by Susan Sontag. The two men inhabit a world where literary lions still roam, now a dwindling tribe loosely connected by the New York Review of Books, where both are longtime contributors. Mr. Pinckney was in Berlin working with Robert Wilson, the avant-garde director, on a production of “Orlando,” the Virginia Woolf novel, which Mr. Pinckney had adapted.

Mr. Pinckney was once a student of Elizabeth Hardwick’s, a co-founder of the NYRB. He is the author of two acclaimed coming-of-age novels, “High Cotton,” out in 1992, and its bookend, “Black Deutschland,” out in 2016. Each features an ambivalent and lettered gay Black protagonist working out his place in the world while untangling the aspirations and complications of his class, sexuality and race. In both books, the narrator flees to Europe, long a more welcoming home to America’s Black intelligentsia.

One night in Berlin in 1990, Ms. Sontag broke her dinner date with Mr. Pinckney to dine instead with Mr. Fenton, who was there on a visit. The evening did not go well. Ms. Sontag was annoyed because the poet ignored her by reading a newspaper, though it was an attempt to make Ms. Sontag talk to the star-struck young photographer he had brought to meet her. The next night, she renewed her dinner date with Mr. Pinckney. Mr. Fenton was in the restaurant too.

“I could tell something had not gone her way,” Mr. Pinckney said. “She said, ‘Look, there he is.’ And there he was. It took a while, but I managed.”

‘A Place out of Time’

They have worked on their house in stages. Mr. White, the architect, first had to make it habitable with plumbing, electricity and a heating system. He had to build their library, a modernist aerie he carved out of a two-story addition built in the 1920s when the place was a medical facility. The top floor is lined with Vitsoe shelving, the state of the art system conceived by Dieter Rams, the German industrial designer. Mr. Fenton bought the rest of the shelves from Archivia, the beloved art and design bookstore on Lexington Avenue that closed in 2012.

It is a working library, arranged in rough alphabetical order and language of origin — books first written in Russian, English, German and so forth. There are satellite libraries in other rooms: decorative arts and design in one room, gardening books in another; and in each man’s study, where books lie in piles on tables, on the arms of sofas and in boxes. In the closets, more books.

Seven years ago, Jane Warrick, a decorative painter, began creating friezes and faux finishes in various rooms. Mr. Fenton would show her a photograph of something he was inspired by — a room with Moroccan mosaic, for example — but mostly he left her alone. For the garden books library, a small room on the third floor Mr. Fenton had painted a bright jungle-y green — his colors are as sensual as his poetry — he brought her an image of an eagle with a snake in its talons, liking what it represented — knowledge subduing ignorance — and so she made a stenciled motif from that.

For three months, she painted faux bois in the oval dining room. “Sometimes James would be reading in the yellow room while I worked,” she said. “He would buy wonderful bits of fabric that were just draped over things. One summer day, I took a video in one of the bathrooms. The window was open and the curtain was sort of breathing with the air. ”

Mr. Amis said: “When I first saw it, it looked like a house made up of ballrooms. James had to tame it and colonize it. It seemed to expand from the center. Every time we went there, civilization would have crept into other bits of the house until he got it up to his standard.”

Shelley Wanger, senior editor at Penguin Random House, and another NYRB alum, recalled book parties for friends like Zadie Smith, and chamber music concerts in the yellow living room, with its pair of pianos, or in the reception hall. Mr. Pinckney said musicians have told him the rooms have perfect acoustics.

“It always felt like a place out of time,” Ms. Wanger said.

Ms. Smith gave the house a cameo in her 2016 novel, “Swing Time.” In it, Mr. Fenton and Mr. Pinckney appear as themselves, befriending the book’s narrator and regaling her, as Ms. Smith writes, with “a long and dramatic story of a house they’d bought, in Harlem, an Edith Wharton-era wreck, which they were doing up with their life savings.”

Later she writes: “I watched them chip away at plaster to reveal original cornices, and fake porphyry by flicking specks of paint at a dark pink wall. Each time I visited, I was moved: by how happy they were together, after so many years! I didn’t have many other models of that idea. Two people creating the time of their own lives, protected somehow by love, not ignorant of history but not deformed by it, either.”

Mr. Fenton finished his tour and served tea in the kitchen, the real living room of the house. A 14-foot table, its top harvested from an old farmhouse in Hudson, was strewn with books and papers. Dishes and pantry items were stored in a row of disparate cupboards and bureaus and a massive 19th-century display cabinet from a department store Mr. Fenton had found on eBay.

On the walls, “family” photographs: a pair of living rooms that were the book-lined lairs of Barbara Epstein, a NYRB co-founder, and Ms. Hardwick, from “New York Living Rooms,” Dominique Nabokov’s 1998 book of portraits of rooms of Manhattanites of a certain kind of renown.

A pair of Annie Leibovitz photographs of the Schaubühne theater in Berlin, which first presented Mr. Wilson and Mr. Pinckney’s production of “Orlando.” A large silk screen print by Tom Phillips, the British artist, blown up from an antique photograph of two men in uniform, one Black, one white, dated 1919, the year of violent race riots in Wales and England, and taken in Aberdeen. One man’s hand rests lightly on the other’s, a gesture startling in its relaxed intimacy.

“I like to think they won,” Mr. Pinckney said.

Mr. Fenton put a plate of cookies on the table. “What we’d like to find is somebody to drag it over the finish line,” he said of the house. “There’s not much left to do.”