It was easy for some to jump on the Brexit result and use it to make a land-grab for Northern Ireland, and it was counterproductive.