The Sun is a main-sequence star, and thus generates its energy by nuclear fusion of hydrogen nuclei into helium. In its core, the Sun fuses 620 million metric tons of hydrogen each second.
This energy is radiated to space.
The sun radiates energy in all directions. Most of it dissipates into space, but the tiny fraction of the sun's energy that reaches Earth is enough to heat the planet and drive the global weather system by warming the atmosphere and oceans. The delicate balance between the amount of heat Earth receives from the sun and the heat that Earth radiates back into space makes it possible for the planet to sustain life.
So the answer is yes, the heat we feel while walking in afternoon is due to nuclear fusion in sun