Cats are notoriously fickle. Regular feeding at a different house might be enough to tempt them away, insensitive to your own feelings of attachment.