The correct option is C ab =1
Tr+1 in the expansion of [ax2+1bx]11
Tr+1=11Cr(ax2)11−r(1bx)r
=11Cr(a)11−r(b)−r(x)22−3r
⇒22−3r=7
⇒r=5
So, coefficient of x7 in the expansion of [ax2+1bx]11 is 11C5a6b−5 ......(1)
Again Tr+1 in the expansion of [ax−1bx2]11 is
Tr+1=11Cr(ax)11−r(−1bx2)r
=11Cra11−r(−1)r×(b)−r(x)−2r(x)11−r
⇒11−3r=7
⇒r=6
Coefficient of x−7 in the expansion of [ax−1bx2]11 is 11C6a5×1×(b)−6 ....(2)
According to given condition , eqn (1) and (2) are equal
11C5a6b−5=11C6a5×1×(b)−6
⇒ab=1