f(f(x))=x2−x+1
Replace x by f(x)
f(f(f(x)))=(f(x))2−f(x)+1
⇒f(x2−x+1)=(f(x))2−f(x)+1 …(1)
Put x=0 in equation (1)
f(1)=(f(0))2−f(0)+1 …(2)
Put x=1 in equation (1)
f(1)=(f(1))2−f(1)+1
⇒(f(1)−1)2=0
⇒f(1)=1
Put f(1)=1 in equation (2), we get
f(0)=0 or 1
But f(0)=0 is rejected because it does not satisfy f(f(x))=x2−x+1
∴f(0) = 1 and f(1)=1
Now, tan−1(1)+sec−1(1)+cos−1(1)+cot−1(1)
=π4+0+0+π4=π2⇒a+b=1+2=3