The correct option is B a=−2,b=−1
We have ,
A=⎡⎢⎣12221−2a2b⎤⎥⎦⇒AT=⎡⎢⎣12a2122−2b⎤⎥⎦∴AAT=9I3⇒⎡⎢⎣12221−2a2b⎤⎥⎦⎡⎢⎣12a2122−2b⎤⎥⎦=9⎡⎢⎣100010001⎤⎥⎦⇒⎡⎢⎣90a+2b+4092a+2−2ba+2b+42a+2−2ba2+4+b2⎤⎥⎦=⎡⎢⎣900090009⎤⎥⎦⇒a+2b+4=0,2a+2−2b=0 and a2+4+b2=9⇒a+2b+4=0,a−b+1=0 and a2+b2=5
Solving a+2b+4=0 and a−b+1=0, we get
a=−2,b=−1.
Clearly, these values satisfy a2+b2=5.
Hence, a=−2 and b=−1.