On Ghana's salt-swept coast, three scaffold-encased towers loom above the Atlantic. Soon they will become luxury apartments, sold at luxury prices – $500,000 (£309,000) for a first-floor flat, with use of a walking track, tennis court, basketball court, swimming pool, gymnasium, steam and sauna room, and cafe bar. A sky-view apartment, one grade above the penthouse, will set you back $2m.

You can only imagine the views of the ocean from the 18-storey apartment blocks. Not all the views, however, are of the ocean. Windows on the north side of the building offer a birdseye view of a neighbouring community, the La slum.

The La traditional fishing community, mainly populated by the Ga ethnic group, is a bustling, ramshackle place, famous for its colourful "fantasy" coffins, but also for being chronically overpopulated, with few toilets, open sewers, and inadequate housing.

Ghana may have enjoyed one of the fastest rates of economic growth in the world in recent years, but for many residents, things are getting worse. The housing deficit stands at 1.7m units. About 7 million more Ghanaians lack access to sanitation than in 1990 and, as a result, there are regular outbreaks of cholera in urban areas.

This week, the government announced it would build 9,120 units of affordable housing, after securing a loan agreement with Barclays and Credit Suisse. It is the kind of project that is welcomed by public sector workers such as nurses and civil servants, who have a reliable income but need homes. But given that the government needs to build, by its own reckoning, 85,000 homes annually for the next 20 years, it is a small dent in a massive problem.

And there is an even bigger issue. According to some estimates, 80% of Ghana's urban dwellers live in slums. At about $20,000 each, affordable housing units are not going to be affordable to many.

The deputy minister for water, works and housing, Sampson Ahi, admitted that regenerating the country's slums was a complex problem, one his government would be unable to solve during its four-year term. "The government is very seriously trying to tackle the slum situations, but I cannot give a specific time by which we will have redeveloped Ghana's slums, because of the economic difficulties that we have, and because of the huge housing deficit," he said.

"So long as people are still moving into the capital because of unemployment in their areas, they don't have anything to do, and every day they keep on coming, so this is not something we can say will be eradicated."

The government says it is working on several housing projects, including public-private partnerships and incentives for the private sector to develop slum areas. These inducements include access to government land, five-year tax holidays, and suspending duty on importing materials to encourage construction of high-rises to accommodate slum dwellers.

If this government manages to improve life for the millions who live in Ghana's growing slums, it would be the first to do so. Previous administrations failed to keep up, even in part, with the housing needs of its people; attempts to build state housing were abandoned by changing governments, poorly managed and corrupt.

The government is relatively open about the extent of the challenge. "The problem that we are facing now with regards [to] our attempt to develop the slums has to do with cultural issues," Ahi said. "People are so attached to their ancestral homes; they have lived there so many years, their parents died there. Demolishing that structure is something that it is difficult to accept. So we are trying to educate and convince them that it will be in their own interests."

"And I fear that if slum dwellers are moved, after the reconstruction they will lose their place. If you are not careful, the newly redeveloped properties will be occupied by the rich."