Some early CPUs (e. g. INTEL 8086) did not implement a mode bit so the operating system and the user applications execute in the same mode of operation.

Explain, if possible, how the system resources could be protected in such architecture.

For processor without kernel mode, it is the duty of application developer to implement all possible checking and follow a general protocol so that the system does not crashes.

Or it would be the duty of the operating system to monitor all instruction issued by the application program. But without hardware level mode settings, OS can be tricked easily. So while using processors without kernel mode it is better to use application only from trusted developer/organization.