The correct option is B 7π8a5
Given, I=2a∫0x3√2ax−x2 dx
⇒I=2a∫0x3⋅x12√2a−x dx
Let, x=2asin2θ
⇒dx=4asinθcosθdθ⇒I=π/2∫0(2asin2θ)72⋅√2acosθ⋅4asinθcosθdθ⇒I=64a5π/2∫0sin8θ⋅cos2θdθ⇒I8,2=64a5⋅(7⋅5⋅3⋅1)(1)10⋅8⋅6⋅4⋅2⋅π2{∵Im,n=(m−1)(m−3)(m−5)⋯(n−1)(n−3)(n−5)(m+n)(m+n−2)(m+n−4)⋯⋅π2, when both m,n is even}I=7π8a5