Let I=∫π20sin2xtan−1(sinx)dx=∫π202sinxcosxtan−1(sinx)dx
Also, let sinx=t⇒cosxdx=dt
When x=0,t=0 and when x=π2,t=1
⇒I=2∫10tan−1(t)dt ............ (1)
Consider
∫t⋅tan−1tdt=tan−1t⋅∫tdt−∫{ddt(tan−1t)∫tdt}dt
=tan−1t⋅t22−∫11+t2⋅t22dt
=t2tan−1t2−12∫t2+1−11+t2dt
=t2tan−1t2−12∫1dt+12∫11+t2dt
=t2tan−1t2−12⋅t+12tan−1t
⇒∫10t⋅tan−1tdt=[t2⋅tan−1t2−t2+12tan−1t]10
=12[π4−1+π4]
=12[π2−1]=π4−12
From equation (1), we obtain
I=2[π4−12]=π2−1