The magnetic property of a material depends on it's electronic configuration.
The electrons in an atom can be either paired or unpaired, when the orbitals are filled by the Pauli exclusion principle.
The e- “spins” around its axis as well as around the nucleus.
A moving charge gives rise to an associated magnetic field.
Atoms which have at least one unpaired electron are termed as paramagnetic and others are diamagnetic.
All paramagnetic materilas are weakly attracted in a magnetic field. But “magnetic materials” which we can sense being attracted by magnets are ferromagnetic which is an extreme case of paramagnetism.
The moving electron has a magnetic moment associated with it.
A group of such atoms give rise to magnetic domains.
If these domains are close enough to interact with each other, then they may either strengthen (if e- spins are along same direction) or cancel out (else).
If the domains aid each other, the material is likely to be magnetic in nature, and non-magnetic otherwise.
So, whether a material is influenced by a magnet simply depends upon it's electron spins and magnetic moments.