Martha’s Vineyard residents privately admitted to not wanting migrants on the island.