WHY do you think a person who wasn't a patriot or a loyalist was sometimes called a fence sitter

I would imagine that this analogy stemmed from the fact that a fence normally serves as a border between two territories. With the patriots being one territory, and the loyalists being the other, one who wouldn't claim either would be left sitting on the metaphoric fence.

A person was called a fence sitter because the person can't decide which side they are on, nowadays we call people who can't make a choice fence sitters. They sit back and watch what happens never committing