Let the lines represented by ax2+2hxy+by2=0 be y=m1x and y=m2x
Then m1,m2 are the roots of the equation bm2+2hm+a=0
m1+m2=−2hb.......(i)m1m2=ab........(ii)
According to the problem the lines represented by ax′2+2hx′y′+by′2=0
are y=m1x and y=−1m2x
m1+(−1m2)=−2h′b....(iii)
m1.(−1m2)=a′b′....(iv)
Multiplying (ii) and (iv)
⇒m21=−aa′bb′m1=√−aa′bb′bb′
Substituting in (ii)
√−aa′bb′bb′.m2=ab⇒m2=ab′√−aa′bb′=ab′√−aa′bb′−aa′bb′=−√−aa′bb′a′b
Substituting values of m1 and m2 in (i)
√−aa′bb′bb′−√−aa′bb′a′b=−2hb√−aa′bb′(1b′−1a′)=−2h12√−aa′bb′=ha′b′a′−b′......(v)
Substituting values of m1 and m2 in (iii)
√−aa′bb′bb′−√−aa′bb′ab′=−2h′b′12√−aa′bb′(1b−1a)=−2h′⇒ha′b′a′−b′=h′aba−b........(vi)
From (v) and (vi)
h′aba−b=ha′b′a′−b′=12√−aa′bb′
Hence proved.