"We just talk about it, like, 'What the f--- do people... What are they thinking?' " the comedian tells PEOPLE of the initial ...