Fox News regular is growing alarmed that some conservatives might not like Black people