Gordon Thornton, 71, claims he gave 'RHOP' star Mia, 38, a hall pass to sleep with other men due to his age