One calorie is defined as the heat required to raise the temperature of 1 g of water from 14.5o C to 15.5o C. Also, exactly 4.186 J of work is to be done on heating 1 g of water from 14.5o C to 15.5o C. So, we cannot round-off 4.186 J to 4 J as it will give significant difference in work done required to raise the temperature and heat required to raise the temperature.
Since work and heat are equivalent here, so on taking 1 calorie = 4 J, the validity of the equivalence of work and heat will get defied. Thus, just to make conversion easy, 1 calorie cannot be taken equal to 4 J.