Since the radius of the cavity is half the radius of the sphere it was made in, the mass removed to make it is one-eighth of the mass of the sphere.
Gravitational Potential Energy =PEsys
⇒PEsys=PEdue to sphere−PEdue to removed mass
PEdue to sphere=−GMm4RPEdue to removed mass=−Gm(M8)4R−R2=−GMm28R.
So, PEsys=−GMm4R+GMm28R=−6GMm28R
Therefore, λ=6