The line suggests that the boy had the ball for a long time. When the ball fell into the harbour, he also lost all his childhood memories of playing with that ball. Buying a new ball would not bring back these memories. This is why the boy was so saddened by losing the ball.