Ezra was born in Cambridge. His mother, Ruth Stiles—daughter of the president of Yale—died when he was seven. When he graduated from Harvard College and Divinity School, William Ellery Channing knocked on his door, whereupon he became Channing's colleague and then his successor. He was minister when the church moved from Federal Street to Arlington Street, opposite the Public Garden. Gannett was an excellent extemporaneous Unitarian preacher in this commanding post.

An organizer, Gannett not only fostered the Unitarian Association but its administrative officer function. He supported the association's ministry-at-large to the Boston poor. However, after twelve exhausting years, he escaped to Europe for two years of recovery. Soon after his return a paralytic stroke left him a cripple who ever after adeptly used two canes to walk. Despite his limp he preached, coedited the Christian Examiner, and opposed the Transcendentalist emphasis on immediate inspiration and the immanence of God affirmed by Emerson and Parker. Harvard named him a Doctor of Divinity.

Although Ezra Gannett hated slavery, he prayed, "God save us from disunion!"