Portuguese power declined in India as a combination of economy and war. As a small nation it did not have the economic resources to maintain their vast colonial holdings in Asia, Africa and South America. Over time they would lose some of their colonies but not all.
In India they lost some of their colonies due to wars against local rulers and their intolerant religious policies that led to insurrections against them. Until 1961, their colonial holdings would only comprise Goa, Diu and Daman and Mahe.
The Dutch in India would lose their colonies as a result of defeat at the hands of the local rulers such as the one against Travancore during the battle of Colachall. The rise of the British also led to decline in Dutch presence in India.
For further reading check the following articles: