The magnitude of gravitational force depends on the masses of two objects and the distance between them.
Because the Earth has the greater mass, all other objects on the Earth are pulled towards it. Also, there exists gravitational force between us and any building. However, because of the smaller mass of the building as compared to the earth, this force is very small to experience. Therefore, we are not pulled towards the building.
also note that G=6.67×10−11Nm2Kg2 , so for considerable amount of force, masses must be large enough, ex: mass of celestial bodies.