The correct option is D (−∞,tan17]
Given : [tan−1x]2−3[tan−1x]2+12>0
We know, −π2<tan−1x<π2
Let t=[tan−1x], then inequality becomes
t2−3t2+12>0
⇒(t−12)(t−1)>0
⇒t<12 or t>1
⇒[tan−1x]<12 or [tan−1x]>1
⇒−π2<tan−1x<1 or tan−1x≥2
⇒−∞<x<tan1 or x∈ϕ [∵−π2<tan−1x<π2]
∴ Set of values of x is (−∞,tan1)
And (−∞,−tan1]⊂(−∞,tan1)
also (−∞,tan17]⊂(−∞,tan1)