The rate of change in momentum when you hit the wall is more than when you hit a piece of sponge with an equal force.
The rate of change in momentum is directly proportional to the force applied. Hence, it hurts more when you hit the wall rather than when you hit a piece of sponge with an equal force.
It comes directly from Newton's second law of motion.