A binary operator * is closed on a set S if for every a,b € S, a*b is also an element of S.
Here A is the set of all real numbers except 0 and 1.
Let's take a = 2 and b = 5
a O b = a + b - ab
= 2 + 5 - 10
= -3 € A
Let's try with other values a = 100 and b = -100
a O b = 100+(-100)-(-1000)
= 1000 € A
Hence a O b does not result in 0 or 1 for any values in A. Hence A is closed under the operation O.