The correct options are
C infinite number of nontrivial solutions for one value of λ
D only one solution forλ≠−5
The determinant will be,
∣∣
∣∣65λ3−1412−3∣∣
∣∣
By, R1=R1−2R2 andR2=R2−3R1
=∣∣
∣∣07λ−80−71312−3∣∣
∣∣=91+7(λ−8)
=7λ+91−56=7λ+35
=7(λ+5)
So, if this is 0,
i.e,λ=−5 it will have infinitely many solutions
and
if λ≠−5, it will have one solution.