Thousands marched against a recent deal he had done to share power with the army, who staged a coup in October.
Chanting “power to the people”, protesters called for a return to full civilian rule. But military forces staged another violent crackdown, leaving two people dead.
Mr Hamdok’s decision to quit leaves the army in full control.
It is another blow to Sudan’s fragile attempts at a transition to democratic rule after a popular uprising led to the overthrow of Sudan’s long-term authoritarian President Omar al-Bashir in 2019.
In a televised address, Mr Hamdok said the country was at a “dangerous turning point that threatens its whole survival”.
Civilian and military leaders had made an uneasy power-sharing agreement after the army staged a coup on 25 October and initially placed Prime Minister Hamdok under house arrest.
Under the agreement reached with Mr Hamdok in November, the reinstated prime minister was supposed to lead a cabinet of technocrats until elections were held. But it was unclear how much power the new civilian government would have, and protesters said they did not trust the military.
Thousands of people were on the streets of the capital Khartoum and the city of Omdurman on Sunday, chanting and calling on the military to leave politics alone.
On social media, activists have said 2022 will be “the year of the continuation of the resistance”.
More than 50 people have been killed at protests since the coup, including at least two on Sunday, according to the pro-democracy Sudan Central Doctors’ Committee.
Analysis by Emmanuel Igunza, BBC News
January 1 marked Sudan’s Independence Day but there’s little to celebrate in the country at the moment.
The resignation of PM Abdalla Hamdok is a big blow to the military leaders who had thought an agreement with Mr Hamdok would appease protesters and legitimize their stay in power.
Clearly those calculations were wrong. But it means the army are now firmly in power, reversing gains made as the country attempted to return to civilian rule.
The current political crisis now threatens to return Sudan to the authoritarian years of former ousted leader Omar al-Bashir.
And there’s also the risk that the country could go back to being a pariah state with the likes of the US already indicating that they would sanction those impeding a return to civilian rule.