If you know the direction of flow of current, can you say all the electrons are going only in that direction at any point of time?
Not every time, but definitely sometimes.
When we say that there is a current of balls, we actually mean that the net movement of the balls over a period of time is in the direction of the current. The balls, individually, could be moving in any random direction at any given point in time.