The correct option is D m−1m−n
Substitute sinx=t⇒cosxdx=dt,
I=∫cosmxsinnxdx=∫cosm−1xcosxsinnxdx
After substituting, we get
I=∫(1−t2)m−12tndt
(1−t2)m−12=tm−1(1t2−1)m−12
Therefore ,
I=∫tm−1(1t2−1)m−12tndt
⇒I=∫tm−n−1(1t2−1)m−12dt
Take (1t2−1)m−12 as the first function, tm−n−1 as the second function and proceed by using integration by parts.
Therefore,
I=(1t2−1)m−12∫tm−n−1dt−∫⎛⎝ddt(1t2−1)m−12∫tm−n−1dt⎞⎠dt
⇒I=(1t2−1)m−12tm−nm−n−∫⎛⎜⎝(m−1)2(1t2−1)(m−3)2(−2t3)tm−nm−n⎞⎟⎠dt
Substitute t=sinx⇒dt=cosxdx,
I=(1(sinx)2−1)m−12(sinx)m−nm−n−∫⎛⎜
⎜⎝(m−1)2(1(sinx)2−1)(m−3)2(−2(sinx)3)(sinx)m−nm−ncosx⎞⎟
⎟⎠dx
I=((cosx)2(sinx)2)(m−1)2(sinx)m−nm−n+∫⎛⎜
⎜⎝(m−1)((cosx)2(sinx)2)(m−3)2(1(sinx)3)(sinx)m−nm−ncosx⎞⎟
⎟⎠dx
On simplification, I=cosm−1x(m−n)sinn−1x+m−1m−n∫cosm−2xsinnxdx
On comparing, A=m−1m−n