At its height, Rome's empire stretched right along the coast of north Africa and sub-Saharan Africans passed to and fro across its porous southern border.