Grandfather says he "regrets it a bit" now that a highway was built around his house