The correct option is C (1+a2+b2)3
∣∣
∣
∣∣1+a2−b22ab−2b2ab1−a2+b22a2b−2a1−a2−b2∣∣
∣
∣∣
R1→R1+bR3;R2→R2−aR3
∣∣
∣
∣∣1+a2+b20−b(1+a2+b2)01+a2+b2a(1+a2+b2)2b−2a1−a2−b2∣∣
∣
∣∣
=∣∣
∣∣10−b01a2b−2a1−a2−b2∣∣
∣∣(1+a2+b2)2R3→R3−2bR−1+2aR−2
=(1+a2+b2)2∣∣
∣∣10−b01a001+a2+b2∣∣
∣∣=(1+a2+b2)3