The correct option is B λ=3,μ≠10
Transformation matrix is ⎡⎢⎣11161231012λμ⎤⎥⎦
applying R3→R3−R2
⎡⎢⎣11161231000λ−3μ−10⎤⎥⎦
when λ=3 and μ≠10 rank of coefficient matrix is less than transformation matrix.
i.e system of equations has no solution
∴ λ=3 and μ≠10