The correct option is A I=2tan−1√x2+x+1x+C
Let I=∫(x−1)(x+1)√x3+x2+xdx=∫(x2−1)(x+1)2√x3+x2+xdx
=∫x2(1−1x2)(x2+2x+1)√x3+x2+xdx
=∫x2(1−1x2)x(x+2+1x).x√x+1+1xdx
Put x+1x=t, (1−1x2)dx=dt
=∫dt(t+2)√t+1, which reduces to ∫dxP√Q.
Put t+1=z2
I=∫2zdz(z2+1)√z2=2∫dzz2+1=2tan−1(z)+C
=2tan−1(√t+1)+C=2tan−1√x2+x+1x+C