What Remains tells the story of a family of German Jews who have escaped from the Nazis a few years before escape became impossible. Karl, his wife, Julia, and his brother, Gustave, have made successful lives for themselves in London, the first brother in business and the second as an art dealer, and they live in comfort even during the worst of World War II. These are people who recite Goethe at the dinner table, play Mozart's chamber music together in the evening, and have a Kollwitz hanging on the wall. Although most of the book takes place in England during the last years of the war and the years immediately following, the family's immigration to America at least the move by Karl, Julia, and their two sons, Jacob and Ben is the action, set offstage, that gives poignancy to what is remembered. Karl defines the larger meaning of these moves in terms of his heritage: "In 1630 his family left Italy; in 1670 his ancestors were driven from Vienna and traveled on to Hamburg and resided there in comfort until Hitler threw them out. It isn't a question of whether but when: death and displacement will come."
The novel is framed by first-person chapters where Ben (who sounds suspiciously like Delbanco himself, even down to the names of his wife and daughters) returns to London to revisit the home where he lived as a very young child. When there, he is overwhelmed by memory: "For some time I wander around the locked house, full of nostalgia and what I can only call Proustian remembrance: this is the corner where that happened, here is the window I rubbed at to peer through the chill wintry fog." Later he finds that he has visited the wrong house; his family actually lived a bit down the street. That uncertain certainty of memory gives this book its center; it also makes the book a "novel."