The correct option is A G.M of a,b
f(x)=x33−abx
f′(x)=x2−ab
Now, for maximum or minimum,
f′(x)=0
Or
x2−ab=0
Or
x2=ab
Or
x=±√ab
Now
f′′(x)=2x
At
x=−√ab
f′′(x)<0.
Hence f(x) attains a maximum at x=−√ab.
Similarly
f′′(x)>0 at x=√ab.
Hence f(x) attains a minimum at x=√ab.
Therefore the function attains a minimum at
x=√ab which is the GM of a and b.