The correct option is D An ellipse with minor axis along [11]
P = [3113]
then,
⇒ [ab] = P [xy]
⇒ [ab] = [3113] [xy]
⇒ 3x + y = a and x + 3y = b
Now, a2+b2 = 1
⇒ (3x+y)2+(x+3y)2 = 1
or 10x2+10y2+12xy = 1 .....(i)
Now comparing it with ax2+by2+2hxy=1, we get
a = 10, b = 10, h = 6
∵ h2 -ab = (6)2 - (10)(1) = -64 < 0
So (i) represents an ellipse with major axis is along y = -x = [1−1] and minor axis is along y = x = [11] So, option (d) is correct.