There's been a similar debate in the UK. My feelings are derived in my own familys past. My family were mill workers in Lancashire, working in atrocious conditions for a pittance. In short they were exploited by wealthy mill owners who were part of a larger system that eventually leads back to the aristocracy. The money made on the back of slavery in the sugar industry and in the textiles industry is still actually held by those families that perpetrated the deeds 200 years ago, although it is disguised via corporate developments and diversification. My family, though NOT slaves and though NOT suffering the terrible experiences visited upon those poor people were part of the same system.
I recognise the argument about racism, but I feel that that is symptomatic of other things rather than a basic perception, generated by the fact of slavery. In short, I don't think I have anything to aplogise for. I think I'm owed an apology from my previous feudal, medieaval and industrial exploiters, but I'm not going to hold my breath. In the meanwhile I'll plot their peaceful downfall.