The ascot connotes informality. It is something one might wear at a cocktail party in one's own flat but is not something you wear out in public.