how do we exactly know that only negative charges flow as we can't see them ? so how do we know that the charges which are flowing are negative can't they be positive?
The flow is because of negative charges because of flow of electrons. Electrons carry negative charge and protons carry positive charge.
On applying electric field, you can expect the protons to move but since electrons are lighter than protons, they will flow and thus negative charge carried by them flows.