Could it be a remnant of humorism, where illnesses were attributed to the imbalance of the humors?

And, naturally, the norm (i.c. being healthy) is decisive for what is considered healthy, and any deviations are unbalanced.

Not all psychological problems are expressed by a person being unbalanced. Using the term to refer to a person suggests volatility or moodiness, ups and downs.

Balance is not just a motionless state, there is dynamic balance which we all experience when we stand and walk. More obviously we balance on a bicycle or a horse and aerialists balance on high wires. The 'balance of the mind' is dynamic maintaining equilibrium among contending psychological forces

healthline.com/health/chemical-imbalance-in-the-brain

Why ask about the precise justification for metaphoric unbalanced in particular? So far as I can see, the alternative terms disturbed and upset as used in OP's cited definitions invoke almost exactly the same metaphoric usage.

Consider: “It’s hard to be level-headed when your head’s not screwed on straight.”

@FumbleFingers Psychological states are described through metaphors because they are intangible and difficult to describe directly. Therefore, people resort to comparisons with material things that are accessible to human senses and are known from experience. Due to the mankind's development, a change in usual lifestyle, a change in scientific views what a metaphor refers to becomes no longer understandable to modern people. Since a basis of a metaphor ceases to be comprehensible, a very concept conveyed through it ceases to be comprehensible. So, I want to clarify the origin of this metaphor.

All language is metaphor. The specific word unbalanced has no special status in relation to [mental] instability - it's just one of many words implying "inability to hold oneself up (against gravity)". That's a pretty natural "defect" for humans to be concerned about, given we're bipedal and wouldn't tend to survive if we couldn't stand up and look around.

"Balance is a motionless state when opposing forces are equal and mutually annihilate each other" this is merely one usage of the word. "unbalanced" obviously means "given to extremes" - there's nothing more to it than that. Questions on here which ask for ultra-profound, epistemological, core-of-reality basis for the "meaning! of! meaning!" are totally unswerable. If OP literally doesn't know the usual many senses of "unbalanced" , simply ask on English Learners.

"A person with psychological problems can be called unbalanced" That sentence is basically wrong. Unbalanced people are called unbalanced. There are any number of psychological problems where the party is not "unbalanced" and the word "unbalanced" wouldn't be used in that case.

Conversely, a party such as, say, an extreme gamer or obsessed "Ultra" style football fan, will be called "unbalanced", with no implication of "psychological problems". They are: unbalanced. Their life is unbalanced, their work hours are unbalanced, etc. Why is there a QA about a completely trivial word meaning?

In modern psychological terminology "unbalanced" might seem unreasonable. Traditionally, and still in layman's terms "unbalanced" is like "off-key" or "off-center", like "out of whack" or "a loose cannon".

@FumbleFingers Please justify your generalization that all language is a metaphor. Some words are metaphors and some words are not. Words that are metaphors mean one thing through another. Most words are not metaphors and denote concepts directly.

@Fivesideddice The statement that balance is the natural state of being is quite contradictory. Everything does strive for balance, but complete balance in nature is death. An unbalance gives rise to movement, as the unbalance of pressure generates wind. The complete balance leads to complete immobility, and complete immobility is death. If we think philosophically, it is an unbalance that makes humans act. For example, the unbalance between what we need and what we have.

@Fattie I am not asking about the meaning of the word, but about its origin or etymology. These are two different things. You explained to me the meaning of the word. The original meaning of balance is scales (an instrument for weighing), that use the principle of mass comparison. Hence the meaning of equality of forces arose. And from here, somehow, people came to the metaphor about the mental state of a person. The meaning of the word you described did not come out of nothing.

@Fattie "Unbalanced" does not imply specifically extreme behaviour, but rather irrational behaviour.

"Interestingly, such comparison exists not only in English but also in Russian (неуравновешенный) and French (déséquilibré)." Out of curiosity, does Chinese also have a word like this? IIRC their traditional medicine system uses the Taoist elements, and imbalances in them were thought to lead to various physical and mental maladies.

@nick012000 Interesting point of view. When I received the first answers about humorism, I also remembered the ideas about the imbalance of elements (e.g. earth, fire, water and air) in a person. But Sven Yargs gave examples of the oldest uses in his second answer. In them, the authors compare an unstable mind to an unbalanced vessel. Now I think this metaphor is most likely associated with an unbalanced vessel, which is underloaded, and therefore unstable on the water. In the past vessels used to be common means of transport. So this metaphor could be understood by a wide range of people.

@Eagle we don’t always think philosophically, and sometimes, in the course of things, we develop misconceptions in our use of language. I didn’t say that balance is the natural state of being, only that we generally think of it in that way. (Besides, isn’t not-alive the natural state of being? Most of the stuff we’re surrounded with is in that particular state most of the time).

Some central elements of Nietzsche's philosophy are; ... all language is metaphor. Admittedly, that's from a long-dead philosopher. But I'd have thought most if not all professional linguists today would agree.