The stone is moving in circle because an external force called the centripetal force is acting on it due to the tension in the string but as soon as the tension is removed the stone carries on its journey in the direction of the velocity at that particular instant, which happen to be in tangential direction because in a circular motion the linear velocity is always tangential to the circle in which the body is moving.