The correct option is D -2
These types of questions are solved using conditional identities.
Here, cos2A+cos2B+cos2C=1+cos2A2+1+cos2B2+1+cos2C2=12(cos2A+cos2B+cos2C)+32Using the conditional identity, when A + B + C = π ⇒cos2A+cos2B+cos2C= −1−4cosA.cosB.cosCwe get,=12(−1−4cosA.cosB.cosC)+32=1−2cosA.cosB.cosC
Now, comparing with a+bcosAcosBcosC
we get,
a=1, b=−2⇒ab=−2.