The correct option is B −1
(tan−1x)2+(cot−1x)2=5π28
Since tan−1x+cot−1x=π2,
So, (tan−1x)2+(π2−tan−1x)2=5π28
⇒(tan−1x)2+π24−πtan−1x+(tan−1x)2=5π28
⇒2(tan−1x)2−πtan−1x+π24−5π28=0
⇒16(tan−1x)2−8πtan−1x−3π2=0
Above equation is quadratic in tan−1x
tan−1x=8π±√64π2+64⋅3π22⋅16
⇒tan−1x=π±2π4
⇒tan−1x=3π4,−π4
⇒tan−1x=−π4
⇒x=−1