We know that it is the electrons that move, so how is the potential defined as the work done to bring a POSITIVE CHARGE from infinity to a point?
See, infinity is the place that is considered to have no charges, and is at 0 potential all the time.
So, potential at a point in an electric field is defined as work done in bringing an unit positive charge from infinite distance to that point. Actually, we are measuring the potential difference between infinity and the required point. But we've named it potential of the point as it is with reference point infinity.
Basically, infinity is considered as a reference place that is fixed. If other points are considered, then one has to define the other point first and the potential at that point to find the potential at a new point.