The correct option is
C Gmm′(x−r)2
The net gravitational field on a point mass
m inside a spherical shell of mass
M is zero, this is because of the symmetric mass distribution outside the position of the test mass
m .
So, the force from any spherically symmetric mass distribution on a mass inside its radius is zero.
Thus, the shell of mass
M exerts no force on mass
m′ as its gravitational field inside is zero.i.e.,
Fshell=0
So the force on the particle will come due to solid sphere of mass
m only.
The point mass
m′ is inside the shell is at a distance
x−r from the centre of the solid sphere of mass
m as shown in the figure.
Field due to the solid sphere at the point where
m′ is placed,
Esphere=
Gm(x−r)2
So, the net force on particle
m′ will be,
Fnet=m′Esphere+Fshell
⇒Fnet=Gmm′(x−r)2+0
∴Fnet=Gmm′(x−r)2
Hence, option (c) is the correct answer.