“He had the distinction of being the only member of the party to have bagged any game...”
The phrase in italics means
(i) Grandfather was the most distinguished member of the party.
(ii) Grandfather was the only sportsperson in the party.
(iii) Grandfather was the only successful member of the hunting party.
The phrase in italics means that the grandfather was the only successful member of the hunting party.