The anti-European Tories were a fringe group - until they took over their whole party.