Why were many members of Congress opposed to Andrew Johnson from the beginning of his presidency?

Andrew Johnson was the acting president after Lincoln's assassination. He was opposed by many members of Congress for various reasons. One of which were his very lenient version of Reconstruction and State Control over voting rights when he assumed office. The second being him favoring slavery. Even though had supported an end to slavery in the 1860's, he was a white supremacist who believed that the country is only for white men and should always be governed only by white men. Which many members of Congress opposed, even when Johnson tried to veto the Civil Rights Bill.