Some conservatives say that whether it's popular or unpopular, imposing strict limits on immigration is the right thing to do, and it must be defended.