Suppose that the upper righthand corner of the rectangle is at the point ⟨x,y⟩⟨x,y⟩.Then you know that the area of the rectangle is, as you say, 4xy4xy,
and you know that
x2a2+y2b2=1 ....... (1)
Thinking of the area as a function of xx, we have
dAdx=4xdydx+4y
Differentiating (1)(1) with respect to xx, we have
2xa2+2yb2dydx=0
dydx=−b2a2xy∴dAdx=4y−4b2x2a2y
Setting this to 00 and simplifying, we have
y2=b2x2a2
From (1)(1), we know that
y2=b2−b2x2a2
thus,
y2=b2−y22y2=b2y2b2=12
Then x2a2=12
and area is maximised when
x=a√2
y=b√2
So, area of greatest rectangle =4xy
Area =4×a√2×b√2
=2ab