What clear evidence of the author’s background do we encounter in The Custom House from The Scarlet Letter?

The narrator wonders whether his Puritan ancestors would scoff at him for wanting to do something as frivolous as writing a book to meditate on human nature.