Comparing the equation
(a2+3b2)x2+8abxy+(b2−3a2)y2=0 with
ax2−2hxy+by2=0, we have,
A=a2+3b2, h=4ab, and B=b2−3a2.
∴H2−AB=16a2b2−(a2−3b2)(b2−3a2)
=16a2b2+(a2−3b2)(3a2−b2)
=16a2b2+3a4−10a2b2+3b4
=3a4+6a2b2+3b4
3(a4+2a2b2+b4)
3(a2+b2)2
∴√H2−AB=√3(a2+b2)
Also, A+B=(a2−3b2)+(b2−3a2)
=−2(a2+b2)
Let θ be the acute angle between the lines.
∴tanθ=∣∣∣2√H2−ABA+B∣∣∣
=∣∣
∣∣2√3(a2+b2)−2(a2+b2)∣∣
∣∣
√3=tan60o
∴θ=60o