The mass of any particle at rest (relative to the observer) is known as the rest mass of that particle. One of the consequences of the Einstein's special theory of relativity is that the mass of the particle increases with increasing velocity relative to the observer, which is guided by the following formula:
m=m0√1−(VC)2
where m is the mass of the particle at velocity v and m0 is the rest mass.
Since the photons are bound to travel at the speed of light (again a consequence of Einstein's special theory of relativity), a non-zero rest mass would lead to an infinitely massive particle (the denominator in the formula becomes zero at v=c), which is not a realistic scenario for physically possible situations. So the only possible value that the rest mass of photon can attain without breaking the formula is a simple zero. All other values simply lead to an infinitely massive particle.