Joseph Benjamin Elms, son of Benjamin John Elms and Ann Jones, was born in Poplar, Middlesex around 1831.

On the 7 May 1861, when working as a quarryman in Victoria, he married Eliza Ann Nelson with whom he had ten children. Joseph and Eliza spent most of their lives in the inner Melbourne suburb of Collingwood.