Fences split up the territory that a hedgehog has to forage in so having a little hole in your fence could well enable it to move in and out of your garden.