Why were many people opposed to african americans enlisting in the continental army?

People back then mostly thought of African Americans as the lowest people and that all their jobs were was to work on farms and just do hard labor for free. So when a African American is doing something other than back breaking labor people get angry. I don't get why because the African is still fighting for the country, but people were kind of loony back then.