Mr Ghedi, who survived several assassination attempts in office, told MPs of his decision after speaking to President Abdullahi Ahmed Yusuf.
Mr Ghedi failed to quell the Islamist insurgency and was blamed for inviting Ethiopian troops onto Somali soil.
In the capital, shells struck for a third day, in the worst fighting for weeks between Ethiopian-Somali government troops and Islamist rebels.
On Sunday, thousands fled the capital,
Salim Aliyow Ibrow - Mr Ghedi's deputy and a close friend - was appointed just moments after Mr Ghedi said he had "proudly" resigned.
The BBC's
She says that despite efforts to salvage his job during talks at the weekend in
In his address to the Somali parliament in Baidoa, Mr Ghedi denied he had come under outside pressure to step down.
"I was not forced to resign, it comes from me. I am not going anywhere and will be here with you as a legislator," he was reported as saying by the Reuters news agency.
Clan rivalries
Mr Ghedi's resignation was swiftly accepted by President Abdullahi Yusuf.
The pair have had a fractured relationship fuelled by clan rivalries during the three years they have worked together in
Mr Ghedi is from the Hawiye clan, which is dominant in
President Yusuf is from the breakaway northern state of Puntland and comes from the Darod clan, the country's second largest.
Observers say the fear is that with Mr Ghedi gone, the Hawiye will now be even more united in their opposition to Mr Yusuf's transitional government.
Aides close to the president said that the resignation was part of a deal to end what he called the political confusion in
President Yusuf told parliament he welcomed the resignation "with respect to the situation the country is undergoing, the humanitarian catastrophe facing us and the longstanding deadlock among us".
The Ethiopians, seen as rivals by many Somalis, have been fighting alongside Somali troops to try and restore order to the fractured country, but many see them as inflaming tensions.
The UN says some 400,000 people have fled the violence in
PHOTO CAPTION
Somali PM Ali Mohamed Ghedi (r) and Somali President Abdullahi Yusuf