The coefficient of x−3 in the expansion of (x−mx)11 is
−792m6
Let x−3 occur at (r+1)th term in the given expansion.
Then, we have:
Tr+1=11Crx11−r(−mx)r
=(−1)r×11Crmrx11−r−r
For this term to contain x−3, we must have:
11-2r =-3
⇒r=7
∴ Required coefficient
=(−1)711C7m7
=−11×10×9×84×3×2m7
=−330m7