Watching talented colleagues walking away from the Cabinet table is never easy.