The correct option is C F is increasing in (0,π2) and (3π2,2π) and decreasing in (π2,3π2)
Given function is
F(x)=∫x0cost(1+t2)dt,0≤x≤2π
On differentiation w.r.t x, apply (Leibnitz rule)
F′(x)=cosx1+x2×1=cosx1+x2, where (1+x2)>0
Here cosx>0 ⇒ x∈(0,π2)∪(3π2,2π)
and cosx<0 ⇒ (π2,3π2)
So, F is increasing in (0,π2) and (3π2,2π) and decreasing in (π2,3π2)