If everything is a little baggy, then everything will look sloppy when you layer one piece on top of another. Your sport coat should easily fit over a shirt and a fine-gauge sweater.