α,β be the roots of ax2+bx+c=0
Then using sum and product of roots, we get
α+β=−baαβ=ca
Now,
1,α+β,αβ are in A.P., so
1+αβ=2(α+β)⇒a+2b+c=0⋯(1)
Also,
1α,12,1β are in A.P., so
1α+1β=1⇒α+βαβ=1⇒−b=c⋯(2)
From equation (1) and (2), we get
a=−b=c
Then the equation becomes,
a(x2−x+1)=0⇒x2−x+1=0 (∵a≠0)
Therefore,
α2+β2−2α2β2α2+β2=1−2α2+β2=1−2(α+β)2−2αβ=3