Druski has already responded to the Diddy party allegations