Checking commutative.
Given : a∗b=(a+b)2∀a,b∈R
∗ is commutative if
a∗b=b∗a
a∗b=a+b2
b∗a=b+a2
b∗a=a+b2
a∗b=b∗a ∀a,b∈R
∗ is commutative.
Checking associative
∗ is associative if
(a∗b)∗c=a∗(b∗c)
Now,
(a∗b)∗c=(a+b2)∗c
=a+b2+c2
=(a+b+2c2)2
=a+b+2c2
And,
a∗(b∗c)=a∗(b+c2)
=a+b+c22
=2a+b+c22
=2a+b+c4
Since (a∗b)∗c≠a∗(b∗c)∀a,b,c,∈R
∗ is not an associative binary operation.