No, Homo habilis were not the only ancestors of human. There were several species that evolved gradually in order from primates in Africa to become Homo spaiens (Humans). All these species are considered as human ancestors. Human evolution is a gradual process that took place slowly in time.
The different stages of human evolution from ancestral primates to modern human beings are as follows:
Dryopithecus and Ramapithecus
Man like primates
Australopithecus(Homo habilis )
Homo erectus
Homo neanderthalensis
Homo sapiens
Dryopithecus and Ramapithecus were two primates that lived around 15 mya and are considered ancestors of human. They were hairy and walked like gorillas and Chimpanzees.Then from them evolved man like primates around 3-4 mya, that were not so tall but walked straight like man.Australopithecus also called Homo habilis evolved around 2 mya. They used some stone weapons and ate fruits. They were human like.Then from them, evolved Homo erectus and then Homo neanderthalensis. Homo sapiens evolved from Homo neanderthalensis around 75,000 -10,000 mya.