Let θ∈(0,π2) and x=X cosθ+y sinθ,y=X sinθ−Y cosθ such that x2+2xy+y2=aX2+bY2 Where a and b are constants then
x2+y2=x2+y2
xy=(x2−y2) sinθ cosθ−XY(cos2θ−sin2θ)
x2+4xy+y2=X2+Y2+2(X2−Y2)sin2θ−2XYcos2theta
=1+2sinθ)X2+(1+2sinθ)Y2−2cos2θ XY
From the question
a=1+2sin 2θ,b=1−2sin 2θ,cos 2θ=0
cos 2θ=0⇒θ=π4, Then a=1+2sinπ2
b=1−2sinπ2
∴a=3,b=−1