The correct option is D Tr(A2B2)=3
Given : A=diag(−1,ω,i)
Using property of diagonal matrix,
⇒A2=diag((−1)2,ω2,i2) =diag(1,ω2,−1)
So, Tr(A2)=1+ω2−1=ω2
and B=diag(1,ω2,−i)
⇒B2=diag(12,ω4,(−i)2) =diag(1,ω,−1)
So, Tr(B2)=1+ω−1=ω
Now,
A2B2=diag((1⋅1),(ω2⋅ω),(−1⋅−1)) =diag(1,ω3,1) =diag(1,1,1)
(∵ω3=1)
So, Tr(A2B2)=3
and AB=diag((−1⋅1),(ω⋅ω2),(i⋅−i))
⇒AB=diag(−1,ω3,1)
So, Tr(AB)=−1+1+1=1