The correct option is D If the product AB is diagonal then, AB=BA
(i) A2=I
⇒A.A=I ∵A is a symmetric matrix
⇒A=A−1 (A is involuntry then, A−1=A)
(ii) AAT=I
⇒A.A=I
⇒A−1=A (A is orthogonal then, A−1=A)
(iii) (AB)T=AB ∵A & B are symmetric matrices
⇒(AB)T=BTAT=BA
If BA=AB then, (AB)T=AB
(AB is symmetric if AB=BA)
(iv) Let product of A and B be ⎡⎢⎣a000b000c⎤⎥⎦
⇒(AB)T=ABBTAT=BA⇒BA=AB
(AB is diagonal matrix, then AB=BA)