The correct option is D -101376
It is know that (r+1)th term (Tr+1) in the binomial expansion
of (a+b)n is given by Tr+1=nCraa−rbr
Assuming that a5b7 occurs in the (r+1)th term of the expansion (a−2b)12 we obtain
Tr+1=12Cr(a)12−r(−2b)r=12Cr(−2)r(a)12−r(b)r
Comparing the indices of a and b in a5b7 and in Tr+1, we obtain r=7
Thus the coefficient of a5b7 is
=12C7(−2)7=12!7!5!×27=12.11.10.9.8.7!5.4.3.2.7!.27=−(792)(128)=−101376