The other week I went to the leaving party of a good friend. Afiyua is heading for a new career in Ghana, as a manager of a small charity that supports orphaned and vulnerable children living in Accra. What struck me about her departure, though, is just how many black Britons I know who have left the UK since the start of the recession.

She had previously been searching for work in the UK; but cuts to the charity and not-for-profit sector meant that looking abroad was her only viable option. Noticing the trend, I did a quick survey of more than 100 black friends and contacts: and I was shocked to find that more than 70% said that, since 2008, they had either left the UK or seriously considered leaving. It seems we are losing black Britons, and this is not a good thing.

What worries me most is that we seem to be losing highly educated people. Around 80% of those I questioned had first, second or even third degrees, and more than two-thirds were between 26 and 35 years old. These are skilled black people who were born and raised in British society. Can the UK really afford to lose skilled workers when what we are most in need of is economic growth?

So why are they leaving? More than 60% said it was to escape the recession and seek business and job opportunities. But almost two thirds felt that the recession had affected ethnic minorities more than white Britons and 94% said the recession had affected them financially or career-wise.

Figures revealed last weekend, showing that unemployment among young black men has doubled in the last three years, gave weight to their fears. According to the Office for National Statistics, more than half of Britain's young black men, and 39% of women, are jobless – more than twice the rate for young white people.

The Runnymede Trust's 2009 bulletin, Race and the Repercussions of Recession, stated that "the onset of the current crash hit black and ethnic minorities harder than the national average worker". This report predicted that unemployment would rise among black and ethnic minorities during the recession, especially for "those whose employers decide not to favour them because in hard times they might be tempted to look after their own".

Rachael, a 22-year-old customer services adviser from London, told me: "In my current job, I noticed there was a higher percentage of black people who were declined a permanent contract after completing their probation… I heard rumours that a person's 'aggressive' attitude was often given as a reason for their contract not being renewed."

Many black Britons will acknowledge the irritation of being considered "aggressive" by their white colleagues. Rachael's experience suggests racial stereotyping and discrimination is still an issue.

Away from the workplace, over the last few years we've seen increased support for fascist groups, with the BNP winning two seats in the 2009 European parliament and increasing its vote by 2% in the 2010 general election; and more recently the far-right English Defence League has been holding anti-Muslim demonstrations.

"More competition for jobs is creating a climate where ethnic minorities have become a scapegoat for white people's circumstances," says Daniel, from south-east London, who left to work in Switzerland in 2011.

It was no surprise that more than a third of the people I asked said they would leave the UK for a country of black origin. For African and Caribbean Britons, black nations offer growing economies, low barriers to entry in business, and a sense of belonging. Louise Darko, who left the UK in 2009, empathises. "They can see various opportunities in their countries of ethnic origin and other black countries, such as entrepreneurship," says the co-owner of MSC Boutique in Johannesburg.

And Rochelle McClymont, a 26-year-old English teacher from London, who works in Korea having lived in Jamaica and Cuba, believes black people "feel we can do better in a country where we are not considered minorities and where our knowledge may be valued more".

But what might seem like a positive transition for those black Britons who are leaving will be a negative one for Britain. If we continue to ignore the growing inequality since the recession began we may not have a skilled black workforce in the near future. Without a rapid and positive change in attitudes towards black Britons, we will continue to lose our black talent, and the diversity that many here have fought so hard to establish. And that will be a loss for all in Britain.

• Follow Comment is free on Twitter @commentisfree