Let three numbers in G.P be a,ar,ar2
Since the sum of the numbers is 21
∴a+ar+ar2=21
⇒a(1+r+r2)=21 .....(1)
Since the sum of the squares of the numbers is 189
∴a2+a2r2+a2r4=189
⇒a2(1+r2+r4)=189 ......(2)
Squaring both sides of (1), we get
a2(1+r+r2)2=441 .......(3)
Dividing (3) by (2), we get
a2(1+r+r2)2a2(1+r2+r4)=441189
⇒(1+r+r2)2(1+r2+r4)=219
⇒(1+r+r2)2(1+2r2+r4−r2)=73
⇒(1+r+r2)2(r2+1)2−r2=73
⇒(1+r+r2)2(r2+1−r)(r2+1+r)=73
⇒(1+r+r2)(r2+1−r)=73
⇒3r2+3r+3=7r2−7r+7
⇒4r2−10r+4=0
⇒2r2−5r+2=0
⇒r=5±√52−4×2×22×2
=5±√25−164
=5±√94=5±34=84,24=2,12
When r=2 from (1)
a(1+2+22)=21⇒a=3
∴ Numbers are 3,3×2,3×22 or 3,6,12
When r=12 from (1)
a[1+12+14]=21
⇒a[4+2+14]=21
⇒a=21×47=3×4=12
∴ Numbers are 12, 12×12, 12×(12)2 or 12, 6, 3