The air strike was the clearest case yet of NATO bombing causing multiple civilian casualties, and comes at a time when NATO is already under strain from a campaign that is taking more time and resources than it expected.
A NATO statement said a military missile site was the intended target of the air strikes but that it appeared one of the weapons did not strike that target.
"NATO regrets the loss of innocent civilian lives and takes great care in conducting strikes against a regime determined to use violence against its own citizens," said Lieutenant-General Charles Bouchard, commander of NATO's operations over Libya.
"Although we are still determining the specifics of this event, indications are that a weapons system failure may have caused this incident," he said.
Reporters taken to the residential area in Tripoli's Souq al-Juma district by Libyan officials early on Sunday saw several bodies being pulled out of the rubble of a destroyed building.
Later, in a hospital, they were shown the bodies of two children and three adults who, officials said, were among those killed in the strike.
Libyan Foreign Minister Abdelati Obeidi said the NATO strike was a "pathetic attempt .... to break the spirit of the people of Tripoli and allow small numbers of terrorists to cause instability and disorder in the peaceful city."
"We will never forgive, we will never forget, we are here; on our land, united with our leader, ready for peace and ready for the fight for our freedom and honor," he told a news conference.
But a spokesman for the rebels fighting to end Muammar Gaddafi's 41-year rule said the Libyan leader was to blame.
"We are sorry for the loss of civilian life that was caused by air strikes carried out by NATO," said Abdel Hafiz Ghoga, vice chairman of the rebel National Transitional Council.
"We hold the Gaddafi regime responsible for having placed its military (installations) near civilian areas," he said. "So these losses are to be expected."
ALLIANCE UNDER STRAIN
NATO has been pounding targets in Libya since March 19 in what it says is an operation to protect civilians who rebelled against Gaddafi's 41-year rule. The Libyan leader says it is an act of colonial aggression designed to steal oil.
Strains are appearing within NATO member states as the campaign drags on for longer than envisaged and Gaddafi remains in power -- even making a show of defiance last week by playing chess with a visiting official.
Defense Secretary Robert Gates said he believed NATO should be allowed to stick at its task. "I think this is going to end OK. I think Gaddafi will eventually fall," he told CNN.
At the scene of the destroyed building on Sunday, clothes, smashed crockery and a rubber duck littered the area.
The building is in a neighbourhood where security forces have in the past few weeks put down anti-Gaddafi protests.
"Why is NATO doing this to us? Why?" asked Ibrahim Ali, who said he lived on the same street as the wrecked building. "NATO is a big problem for the Libyan people. NATO doesn't have any business here, this is between the Libyan people."
Another man, who gave his name as Tony, nodded toward the remains of the building and said: "They (local people) don't like this ... But they don't like the regime either."
