Why did members of the working poor oppose the Emancipation of slaves?

slaves were free labor, so the slave owners profited very heavily off of slaves because they didnt pay for their work. The freedom of slaves, would cost the plantation owners more money because they would now need to pay their employees. The working class people were also scared that the now free slaves would steal jobs away from them.