The correct option is B (cot2,∞)
(cot−1x)2−7(cot−1x)+10>0⇒(cot−1x)2−5(cot−1x)−2(cot−1x)+10>0⇒(cot−1x)[(cot−1x)−5]−2[(cot−1x)−5]>0⇒(cot−1x−2)(cot−1x−5)>0⇒cot−1x∈(−∞,2)∪(5,∞) ⋯(1)
We know that cot−1x∈(0,π) ⋯(2)
So, from (1) and (2)
0<cot−1x<2
As, cot−1x is decreasing function,
⇒x∈(cot2,∞)