The correct option is D 6tan−1(cosx)+2(cosx)−1−cosx+C
Let I=∫sin3x+sin5xcos2x+cos4x
I=∫(sin2x+sin4x)sinxcos2x+cos4xdx
Put cosx=t⇒sinx dx=−dt
I=−∫(1−t2)+(1−t2)2t2+t4dt
⇒I=−∫2−3t2+t4t2+t4
2−3t2+t4t2(1+t2)=1+2−4t2t2+t4
Let 2−4t2t2(1+t2)=At2+B1+t2
⇒A=2,B=6−1
I=−∫(1+2t2−61+t2)dt
⇒I=−(t−2t−6tan−1(t))+C
∴I=6tan−1(cosx)+2(cosx)−1−cosx+C