The correct option is D a=−1,b=1
Given f(x)=⎧⎪
⎪
⎪
⎪
⎪
⎪⎨⎪
⎪
⎪
⎪
⎪
⎪⎩x2a0≤x<1a,1≤x<√22b2−4bx2,√2≤x<∞
Since, f(x) is continuous in 0≤x<∞. So f(x) is continuous at x=1
f(1)=LHL=RHL
Here, f(1)=a
LHL=limx→1−f(x)=limh→0f(1−h)
=limh→0(1−h)2a=1a
∴ 1a=a⇒a2=1
⇒ a=±1
Also, f(x) is continuous at x=√2
LHL=limx→√2−f(x)
⇒LHL=a=±1
f(√2)=2b2−4b2=b2−2b
For a=−1
−1=b2−2b
b2−2b+1=0
b=1
For a=1
b2−2b−1=0
⇒(b−1)2=2
⇒b=1±√2