The correct option is C |A|=0
An n×n homogeneous system of linear equations has a unique solution (the trivial solution) if and only if its determinant is non-zero. If this determinant is zero, then the system has an infinite number of solutions. i.e. For a non-trivial solution |A|=0.