Conservation of energy says that the light energy that gets scattered in other directions is removed from the light that passes through in the forward direction. Since high frequencies are preferentially scattered, they must be depleted from the light that comes straight from the sun.
This is particularly noticeable at sunrise and sunset. Then, the sun is shining at a grazing angle through the atmosphere, so the light passes through an unusually large thickness of air, and the depletion of high frequencies is particularly pronounced. The sun appears yellow, orange or red rather than the usual blinding white.
(In practice, this story can be complicated by dust and smog in the air. However, the sun would redden at sunset even in the absence of dust or pollution.)