Elsgood's grandchildren were the grandchildren not the children of Elsgood, and they could have copied Naruto. They call him Grandpa Elsgood not dad so that means Elgood is their grandpa not their dad.