Potential difference means the differnce in energy of electrons in between two ends.
Now if we connect a battery in a circuit. The negative end will have more electrons with high energy. But positive side will have electrons with less energy (Electrons are present every where. So we cannot say that negative side contains electrons and positive side does not contain electrons). When the circuit is completed we are getting a potential difference between negative side with high energy and positive side with low energy and electrons started to move from negative side to positive side.