Dermot O'Leary was close to quitting the X factor