I was labelled "breaker of marriages" - Agyemang Prempeh