During noon, the Sun's rays fall directly on the Earth's surface. The Sun being a yellow star, its rays also appear yellow at noon. Since yellow light has a high relative luminosity, it produces a high sensation of visibility in our eyes, thereby making the Sun appear brighter.
However, in the morning and evening, due to the slanting rays of the Sun on the Earth's surface, the smaller and middle range wavelength of the rays get scattered in the upper atmosphere. Thus, the sun appears orange during these two times. Since red/orange light has a low relative luminosity, it produces a low sensation of visibility in our eyes, thereby making the Sun appear less brighter.