The correct options are
B a, b, c are in G.P.
C α is a root of ax2+2bx+c=0
The given system of equations will have a non-trivial solution if
Δ=∣∣
∣∣aα+babbα+cbc0aα+bbα+c∣∣
∣∣=0
Applying R3→R3−αR1−R2, we get
Δ=∣∣
∣
∣∣aα+babbα+cbc−(aα2+2bα+c)00∣∣
∣
∣∣=0
⇒−(aα2+2bα+c)(ac−b2)=0
⇒aα2+2bα+c=0orac−b2=0
⇒α is a root of ax2+2bx+c=0 or a,b,c are in G.P.
Hence, option B and D.