Let we have square tiles of a certain length, which we will call ′x′cm.
The area of each tile is x2cm2.
The total area paved would be 600x2 cm2.
According to given question,
600x2=864(x−1)2
⇒600x2=864x2−1728x+864
⇒264x2−1728x+864=0
⇒24(11x2−72x+36)=0
⇒11x2−66x−6x+36=0
⇒11x(x−6)−6(x−6)=0
⇒(x−6)(11x−6)=0
⇒x−6=0,11x−6=0
⇒x=6or611
Hence, the length of tiles=x=6cm.