The correct option is B [cos2x−sin2xsin2xcos2x]
|A|=[1tanx−tanx1]=1+tan2x≠0
So, A is invertible. Also,
adjA=[1tanx−tanx1]T=[1−tanxtanx1]
Now,
A−1=1|A|adjA
⇒A−1=1(1+tan2x)[1−tanxtanx1]
=⎡⎢
⎢
⎢⎣11+tan2x−tanx1+tan2xtanx1+tan2x11+tan2x⎤⎥
⎥
⎥⎦
∴ATA−1=[1−tanxtanx1]⎡⎢
⎢
⎢⎣11+tan2x−tanx1+tan2xtanx1+tan2x11+tan2x⎤⎥
⎥
⎥⎦
=⎡⎢
⎢
⎢
⎢⎣1−tan2x1+tan2x−2tanx1+tan2x2tanx1+tan2x1−tan2x1+tan2x⎤⎥
⎥
⎥
⎥⎦
=[cos2x−sin2xsin2xcos2x]