Given that
(tan−1x)2+(cot−1x)2=5π28
We know that,
tan−1x+cot−1x=π2 .....(1)
Squaring on both sides
⇒(tan−1x+cot−1x)2=(π2)2
⇒(tan−1x)2+(cot−1x)2+2(tan−1x)(cot−1x)=π24 ....(2)
from the given information,
(tan−1x)2+(cot−1x)2=5π28
Substituting this in (2), we get
⇒5π28+2(tan−1x)(cot−1x)=π24
⇒2(tan−1x)(cot−1x)=−3π28 ....(3)
We know that (a−b)2=(a+b)2−4ab
Using this formula, we obtain
(tan−1x−cot−1x)2=(tan−1x+cot−1x)2−4(tan−1x)(cot−1x)
⇒(tan−1x−cot−1x)2=π24−2(−3π28) (from(2),(3))
⇒(tan−1x−cot−1x)2=π24+3π24
(tan−1x−cot−1x)2=π2
Taking square root,
⇒tan−1x−cot−1x=π .....(4)
On solving (1) and (4), we get
⇒2tan−1x=3π2
⇒tan−1x=3π4
⇒x=tan(3π4)
⇒x=−1