Checking commutative.
Given: a∗b=ab2
∀a,b∈Q,ab2∈Q. So, it is binary
∗ commutative if,
a∗b=b∗a
Now, a∗b=ab2
And, b∗a=ba2
Since a∗b≠b∗a
∗ is not commutative.
Checking associative..
∗ is associative if,
(a∗b)∗c=a∗(b∗c)
Now, (a∗b)∗c=ab@∗c
=(a+ab)+(ab2)c2
=ab2c2
And, a∗(b∗c)=a∗bc2
=a+(bc2)2
=ab2c4
Since (a∗b)∗c≠a∗(b∗c)
∗ is not an associative binary operation.