Magnets do not attract all metals, only few special metals (such as iron, cobalt, nickel) gets attracted towards magnet. And these special metals are known as magnetic materials.
Magnetic materials have atoms which contain unpaired electrons . Due to presence of these unpaired electrons each atom of magnetic materials will act like a tiny magnet. And in absence of any magnets, all the magnetic atoms will randomly directed, resulting net magnetisation to be zero.
When a magnet is kept close to such magnetic materials each of the magnetic atom aligned in the same direction according to the external magnetic field. Then the magnetic material acts like a magnet and thus is attracted by the magnet.
So we can conclude due to the presence of tiny atomic magnets, magnetic materials are strongly attracted by the magnets.
In materials like plastic they do not contain unpaired electrons. So the atoms of plastic does not act like individual magnets. Thus when we place them close to magnet, the atoms are not aligned. Hence the substance does not act as magnet and not attracted by the magnets.