The correct options are
A 18
B 12
Let the three numbers in G.P. are ar,a,ar.
Then, ar×a×ar=1728.
⇒a3=1728
⇒a=12
⇒12r+12+12r=38
⇒12(1r+r)=26
⇒12(1+r2)=26r
⇒12r2−26r+12=0
⇒6r2−13r+6=0
⇒6r2−9r−4r+6=0
⇒(3r−2)(2r−3)=0
⇒r=23,32
If r=23, then the numbers are 18, 12 and 8.
If r=32, then the numbers are 8, 12 and 18.