The Asymmetry of Daylight

There is a time of day when the sky is beginning to get brighter; everything is waking up.  There is also a time of day when the sky is beginning to get dark.  There should be a symmetry in the amount of sunlight, and, assuming we could control for environmental factors (such as the number of cars on the road), those two moments should be indistinguishable.

How come we can almost always distinguish between the two?