There is some attractive forces between the particles in the rubber band. This is the reason for the retaining the original position after the stretch.
But the particles are at different distances from each other. (They are not at same distance like in a solid). So the attractive force between two particle is not uniform. This means that the force of attraction between the particles which are far away is weaker and the force of attraction between the particles which are closer are stronger.
When we are stretching the rubber band, the weaker forces are completely broken and we are having only strong forces acting on it. So when we leave the rubber band after the stretch only the stronger forces are bringing the original position. So the original position is not completely restored.