Let the x and y axis be rotated by an angle is as shown below:
then, x=xcosθ−ysinθ
y=xsinθ+ycosθ, where (x, y) is the coordinates with respect to new coordinate axes.
Given: ax2+2hxy+by2+2gx+2fy+c=0
Replace x→xcosθ−ysinθ
y→xsinθ+ycosθ
⇒a(xcosθ−ysinθ)2+2h(xcosθ−ysinθ)(xsinθ+ycosθ)+b(xsinθ+ycosθ)2+2g(xcosθ−ysinθ)+2f(xsinθ+ycosθ)+c=0
⇒a(x2cos2θ−y2sin2θ)+2h(x2sinθcosθ+xycos2θ−xysin2θ−y2sinθcosθ)+b(x2sin2θ+y2cos2θ)+2g(xcosθ−ysinθ)+2f(xsinθ+ycosθ)+c=0
Now taking out every xy term
−2axysinθcosθ+2hxycos2θ−2hxysin2θ+2hxysinθcosθ
To eliminate the xy term, put coefficient of xy=0
⇒−2asinθcosθ+(2hcos2θ−2hsin2θ)+2hsinθcosθ=0
⇒2h(cos2θ)+sin2θ(b−a)=0
⇒tan2θ=2ha−b
⇒θ=12tan−1(2ha−b).