Checking commutative.
Given: a∗b=a2+b2
∀a,b∈Q,a2+b2∈Q. So, it is binary
∗ is commutative if,
a∗b=b∗a
Now a∗b=a2+b2
And, b∗a=b2+a2
=a2+b2
Since a∗b=b∗a ∀a,b∈Q,
∗ is commutative.
Checking associative.
∗ is associative if,
(a∗b)∗c=a∗(b∗c)
Now, (a∗b)∗c=(a2+b2)∗c
=(a2+b2)2+c2
And, a∗(b∗c)=a(b2+c2)
=a2+(b2+c2)2
Since (a∗b)∗c≠a∗(b∗c)
∗ is not an associative binary operation.