You might want a certain type of Brexit deal, but you can't get it if the numbers aren't there.