Let A=[abcd] ⋯(1)
Given Tr(A)=3
⇒a+d=3
⇒d=3−a
Putting in equation (1),
A=[abc3−a]
Also, Tr(A)3=−18
A3=[abc3−a][abc3−a][abc3−a]
⇒A3=[a2+bc3b3ccb+(3−a)2][abc3−a]
⇒A3=[a3+abc+3bca2b+b2c+9b−3ab3ac+c2b+c(3−a)23bc+cb(3−a)+(3−a)3]
Tr(A3)=a3+abc+3bc+3bc+3bc−abc+(3−a)3=−18
⇒a3+9bc+(3−a)3=−18
⇒a3+9bc+27−a3−3⋅3a(3−a)=−18
⇒a2−3a+bc=−5 ⋯(2)
Now, |A|=a(3−a)−bc
=3a−a2−bc=5 [From equation (2)]