Millions of people are falling out of the middle class into the ranks of the poor.