a,b,c are in G.P.
⇒b2=ac
Using sine rule, we have
⇒sin2B=sinAsinC
Multiply both sides by 2 we get
⇒2sin2B=2sinAsinC
⇒2(1−cos2B)=cos(A−C)−cos(A+C)
Given:largest angle exceeds the smallest by 600
⇒A=600+C or A−C=600
⇒2−2cos2B=cos600−cos(1800−B)
⇒2−2cos2B=12+cosB
⇒2cos2B+cosB−2+12=0
⇒4cos2B+2cosB−3=0
⇒cosB=−2±√4+488
But |cosB|<1 so that
cosB=√13−14