In both the cases the potential energy of the spring increases. Since it is given by, and 'x' is measured from the mean position ,i.e., a position at which the spring is neither stretched or compressed, so whichever way we go ,i.e., compress the spring or stretch the spring the 'x' increases and hence the potential energy of the spring.