Consider the given integral.
I=∫x2(tan−1x)1+x2dx
Put
t=tan−1x
dt=11+x2dx
Therefore,
I=∫t.tan2tdt
I=∫t.(sec2t−1)dt
I=∫(t sec2t−t)dt
I=∫t sec2tdt−∫tdt
I=ttan t−∫1.tan tdt−t22+C
I=ttan t−[−log|cost|]−t22+C
I=ttan t+log|cost|−t22+C
Put the value of t, we get
I=tan−1xtan(tan−1x)+log∣∣cos(tan−1x)∣∣−(tan−1x)22+C
I=xtan−1x+log∣∣cos(tan−1x)∣∣−(tan−1x)22+C
Hence, this is the required value of integral.