Among the reasons non-slaveholding whites supported slavery was the

Many non-slave holders, being poor, had aspirations of owning large tracks of land to which would be worked by slaves. Contention with slavery would disrupt this ideal. As long as blacks remained slaves, they would remain at the lowest level of the social order. As such, poor non-slave slave holders would not be at the bottom. This notion would not change as time progressed and even Abraham Lincoln felt that whites were more superior to blacks.