Checking commutative.
Given: a∗b=(a−b)2
∀a,b∈Q,(a−b)2∈Q. So, it is binary
∗ commutative if,
a∗b=b∗a
Now, a∗b=(a−b)2
And, b∗a=(b−a)2
=(a−b)2
Since a∗b≠b∗a ∀a,b∈Q
∗ is not commutative.
Checking associative.
∗ is associative if,
(a∗b)∗c=a∗(b∗c)
Now, (a∗b)∗c=(a−b)2∗c
=((a−b)−c)2
And, a∗(b∗c)=a∗(b−c)2
=(a−(b−c)2)2
Since (a∗b)∗c≠a∗(b∗c)
∗ is not an associative binary operation.