The correct option is
B R√1−8cosAcosBcosCLet O be the Cirumcentre and H be the orthocentre then
OH=√9R2−(a2+b2+c2)
=√9R2−4R2(sin2A+sin2B+sin2C)
=R√9−4(sin2A+sin2B+sin2C)
=R√9−4(1−cos2A+1−cos2B+1−cos2C2)
=R√9−2(3−(cos2A+cos2B+cos2C))
=R√3+2(cos2A+cos2B+cos2C))
=R√3+2(2cos(A+B)cos(A−B)+2cos2C−1)
=R√3+2(2cos(π−C)cos(A−B)+2cos2C−1)
=R√3+2(−2cos(C)cos(A−B)+2cos2C−1)
=R√3+2(2cos(C)(cosC−cos(A−B))−1)
=R√3+4cos(C)(cosC−cos(A−B))−2
=R√1+4(cos(C)(cos(π−(A+B))−cos(A−B))
=R√1−4(cos(C)(cos(A+B)+cos(A−B)))
=R√1−4(cos(C)(2cosAcosB))
=R√1−8cosC.cosA.cosB