What If the Universe Isn't Uniform?