If y = mx be one of the bisectors of the angle between the lines ax2−2hxy+by2=0, then
h(1−m2)+m(a−b)=0
Here equation of one bisector of angle is y - mx = 0, therefore equation of second is x + my = 0.
Hence combined equation is (x + my)(y - mx) = 0
⇒−mx2−xy(m2−1)+my2=0 ....................(i)
Also equation of bisectors of ax2−2hxy+by2=0 is
−hx2−(a−b)xy+hy2=0 ................(ii)
Hence (i) and (ii) are the same equations , therefore
mh=m2−1(a−b)⇒h(m2−1)=m(a−b)
⇒m(a−b)+h(1−m2)=0