The correct option is D a=–3,b=1
Given: (x2−1) is a factor of (x4+ax3+3x−b)
Let f(x)=x4+ax3+3x−b
Using factor theorem, x=±1 are the zeros of f(x).
∴f(1)=0 and f(−1)=0
Taking f(1)=0
⇒a−b+4=0 . . . . .(i)
And f(–1)=0
⇒a+b+2=0 . . . . .(ii)
On solving (i) and (ii),we get
a=–3,b=1