The correct option is
D 13[−23a3√(a3+x3)3+25a3√(a3+x3)3]+C.
∫x5√x3+a3dx
u=x3+a3→dx=13x2du
=13∫(u3/2−a3√u)du
=13[∫u3/2du−a3∫√udu]
=13[u5/25/2−a3×u3/23/2]
=13[2(x3+a3)5/25−a3×2(x3+a3)3/23]
=13[2√(x3+a3)55−a3×2√(x3+a3)33]
∫x5√x3+a3dx
=13[−23a3√(a3+x3)3+25a3√(a3+x3)3]+C.