The correct option is B a2+b2−3
α−β=2θ
cosα+cosβ=a ...(1)
sinα+sinβ=b ...(2)
Squaring and adding eq. (1) & eq. (2)
cos2α+cos2β+sin2α+sin2β+2(cosαcosβ+sinαsinβ)=a2+b2⇒1+1+2cos(α−β)=a2+b2⇒2cos(α−β)=a2+b2−2⇒2cos2θ=a2+b2−2
cos3θcosθ=4cos3θ−3cosθcosθ=4cos2θ−3⇒cos3θcosθ=2(1+cos2θ)−3=2cos2θ−1⇒cos3θcosθ=(a2+b2−2)−1=a2+b2−3
Hence, option 'D' is correct.