We have,
I=∫π20sin2xtan−1(sinx)dx
=∫π202sinxcosxtan−1(sinx)dx
Let
sinx=t
cosxdx=dt
Change limit
sin0=t
t=0
And,
sinπ2=t
t=1
Then,
∫102ttan−1tcosxdx
=∫102ttan−1tdt
=2∫10ttan−1tdt
On integrating and we get,
2[tan−1t∫10tdt−∫10(d(tan−1t)dt∫10tdt)dt]
=2[tan−1t[(t22)01]−∫1011+t2t22dt]
=2tan−1t(t22)01−∫10t21+t2dt
=2tan−1t(t22)01−∫10t2+1−11+t2dt
=2tan−1t(t22)01−∫10t2+11+t2dt+∫1011+t2dt
=2tan−1t(t22)01−∫101dt+∫1011+t2dt
=2tan−1t(t22)01−[t]01+[tan−1t]01+C
=2[tan−11−tan−10][122−022]−[1−0]+[tan−11−tan−10]+C
=2[π4−0][12]−1+[π4−0]
=π4+π4−1
=2π4−1
=π2−1
Hence, this is the answer.