Being Lazy is Rarely in Your Control