How does quartering soldiers affect rights?

People have the right to privacy and refusal to let anyone into their homes under the constitution so the quartering of soldiers would go against those right

In America the third amendment says that a soldier may not live in a home unless given permission from the owner. So I'm not entirely sure how this would effect the rights of a person