The fear must be that the apparently increased competitiveness is less to do with the rest rising than with the big sides collapsing

When Luis Suárez handled that Dominic Adiyah header on the line in the final minute of the World Cup quarter-final between Ghana and Uruguay, he stopped the vibrant Ghanaian team becoming the first African side to reach the semi-final of a World Cup.

For Ghana to do so was what Fifa wanted, probably what the world wanted. They wanted an African side to break new ground at the African World Cup, because that would allow everybody to believe in progress. Or, rather, "Progress". For this isn't a fact, it's a dream, a seductive myth that distorts the narrative so that we can all believe everything's getting better.

In 1978, Tunisia became the first African side to win a World Cup game, beating Mexico. In 1982, Cameroon went home unbeaten, edged out on goals scored by Italy after drawing all three group matches. Algeria beat West Germany, but were eliminated on goal difference after the farce of Gijón as Austria and Germany settled on a result that took both through. In 1986, Morocco became the first African side to reach the second phase.

In 1990, Cameroon beat Argentina in the opening game and so enthralled the world that there are still those prepared to swear Gary Lineker dived to win the two penalties that turned the quarter-final England's way after they'd been outplayed for long spells.

Stop there. For 12 years there had been clear progress, an improvement at each stage. It seemed inevitable that the improvement would continue. Africa had arrived and the assumption was that, sooner or later, an African side would win the World Cup. But 20 years on are we really any closer? The players are clearly there, evidenced by the number of Africans at clubs in the latter stages of the Champions League, but the game at a national level seems to have frozen as it was 20 years ago.

The years of stagnation

Nigeria looked as though they might carry on the improvement in 1994 but, after impressive wins over Bulgaria and Greece, they were a little unlucky to be beaten by Italy in the second round. Morocco and Cameroon had already gone out. In 1998, Nigeria were much fancied again. They beat Spain, but were hammered 4-1 by Denmark in the second round. Cameroon, South Africa and Tunisia didn't manage a win between them.

Senegal began the 2002 World Cup as Cameroon had begun in 1990, by beating the holders. They reached the quarter-final, but after the France victory they didn't win another game in normal time in the tournament. Tunisia, Nigeria, South Africa and Cameroon all went out in the first round. Still, it was a second quarter-finalist: Africa, went Progress's apologists, was consolidating.

By 2006 Africa was widening its breadth of talent. It wasn't just the traditional powers any more. There were new forces such as Ivory Coast, Ghana, Angola and Togo. Ivory Coast got a hideous draw and went out in the first round, but they are a genuine power even if they habitually underperform at the Africa Cup of Nations. Angola, Togo and the perennial first-round losers Tunisia also failed to make it through. Ghana did, signalling the rebirth of their football, but were well-beaten by Brazil in the second phase. Neither Angola or Togo even qualified for the Cup of Nations two years later.

And so to 2010, the first World Cup to be held in Africa, and the first time the African confederation had been granted six berths (the same number, incidentally, as the total number of African teams to have made the second phase). Ivory Coast were undone by a draw that grouped them with Brazil and Portugal. South Africa, Algeria, Cameroon and Nigeria all went out in the first round. Only Ghana made it through.

There was another contentious late handball decision in that World Cup, but it tends to get overlooked. It doesn't have to be recalled to support Progress. It came with two minutes remaining of the final group game between Serbia and Australia. As Tim Cahill jumped in the box, the ball clearly struck the back of his raised arm. It wasn't quite the certain penalty Serbia's coach Radomir Antic claimed it was, but on balance, it probably should have been given. Had Serbia scored that, it would have been they rather than Ghana who made it through to the second phase. So although a handball prevented an African side reaching the semi-finals for the first time, a handball not given prevented a situation whereby no African side made it through to the second phase for the first time since 1982.

Wilful blindness

Is this Progress? No, of course it isn't, nothing like it – and given the quality of player Africa has produced, that is nothing short of disgraceful. Admittedly the African performance at the World Cup might have been better had Egypt, winner of the past three Cup of Nations and significantly the best side on the continent, not twice failed to qualify. And Ivory Coast twice have had awful draws.

But there's also a subjective element. It's easy to be sucked in, to subscribe to the myth. It's easy to make excuses for the football. I'd done it and the moment the scales fell from my eyes was genuinely shocking. I was having a coffee in a courtyard shaded by mango trees in Benguela, western Angola, talking to a Swiss clown who'd married a Nigerian woman he'd met while touring with his circus. He's spent much of the past 20 years writing about African football. "It's all rubbish," he said. "I hate it now, hate what's happened. I hate the lies and the false consciousness. I hate the bullshit and the corruption. Just look at it: it's rubbish."

I instinctively tried to protest, tried to defend African football, but the argument was too simple, too obviously right – sometimes you have to just say what you see and forget all the justifications and theories and agenda. The football in Angola two years ago was rubbish. Only Egypt and Ghana played with anything like cohesion. Yes, Zambia flickered, Malawi were plucky and Gabon were doughty, but there was some miserable football played. Nigeria somehow sulked their way to the semi-final. Cameroon were ramshackle at the back and Samuel Eto'o's lack of faith in his team-mates led to him playing far too deep. Algeria played practically nihilistic football, time-wasting their way to a semi-final – and were allowed to do so by weak refereeing.

Nobody wanted to know. People like the familiar. I won't name him (and it was nobody connected to the Guardian) but one of my editors sent back a column saying it was too negative. He wanted smiling fans with painted faces, drummers and horns, thrillingly muscular forward play and all the other cliches of the African game – and this from a tournament that began with an attack by gunmen on the Togo team bus that left three people dead.

Perhaps there's even guilt on the part of western Europeans, an awareness that there's something distasteful about an economic system that means the best players from Africa, if they're to be properly remunerated, have to move to a different part of the world to perform for western Europe's benefit. That's true of other parts of the globe, of course, but the economic imbalance isn't as stark and, elsewhere, doesn't have the same awkward echoes of colonial exploitation.

Talk of a new slave trade is unhelpfully emotive, but there is an unpleasant traffic in vulnerable and often naive young players, and it seems hard to deny that the demands of the European market have shaped the tactical development of African football. Tom Vernon, who runs an academy near Accra in Ghana and scouts for Manchester United, speaks of the "Pape Bouba Diop" template: having seen the success of big, muscular west African players, clubs go to west Africa looking for more big, muscular players and so that sort of player is prioritised, something that in part explains the dearth of west African creative players in the decade between Abedi Pelé, Jay-Jay Okocha and Kanu and the emerging generation of Kwadwo Asamoah, Dede Ayew and Gervinho.

It even suits the football administrations of individual countries within Africa to pretend that all is well, that things are developing. Not to do so, after all, would be to admit failure, and to do that would be to risk the sinecures that bring wealth, prestige and influence. The myth of Progress is sustained by a conspiracy of the complacent and the self-interested.

A new hope?

The warning signs were there, and were borne out by the World Cup. The traditional powers have reaped what they've been sowing for years. Shambolic and/or corrupt administration coupled with political interference leads to confused governance and an infrastructure that never develops. Sort that out, and improvement can come rapidly, as Senegal have found over the past couple of years. Or look at the case of Kenya, as outlined by Brian Oliver in the third issue of The Blizzard, the surge up the rankings they enjoyed checked by the venality of their football administration.

Or for sheer basic incompetence, it's hard to beat South Africa's failure to qualify for this year's Africa Cup of Nations, playing for a draw against Sierra Leone when they needed a win in their final game because they didn't understand the regulations for separating teams finishing level on points. What made it worse was that the same confusion about three-way mini-leagues led to Gabon celebrating making it out of the group in Angola when actually they'd finished third. So much for the legacy of the World Cup.

That's what makes this year's tournament so fascinating. Egypt's failure to qualify was – in part – down to the uprising against Mubarak, but the absence of Nigeria, Cameroon and South Africa is their own fault. The question is whether the sides who have replaced them are any better. There are three debutants: the co-hosts Equatorial Guinea; Niger, the frankly lucky beneficiaries of South Africa's idiocy; and Botswana, whose comfortable qualification with a largely domestic-based team under the former army officer Stanley Tshosane is genuinely inspiring. Zambia, almost two decades on from the plane crash that wiped out their greatest side, could establish themselves. Senegal have a fearsome strike force. Morocco are on the rise. Guinea deserve credit for taking their chance and eliminating Nigeria. Libya's rise since the war against Gaddafi began is extraordinary.

The positive spin is that there is a greater breadth of talent in Africa than ever before, that there are now a couple of dozen decent sides and that success isn't restricted to the usual five or six nations. When we heard that before, in 2006, it was only half true. Perhaps this time it is, but the fear must be that the apparently increased competitiveness of African football is less to do with the rest rising than with the big sides collapsing, that this isn't Progress, but a descent into mediocrity.