@MPitts It certainly is a proof that $F$ is not a functor, because the demand that $F(f)$ should be a map from $F(A)$ to $F(B)$ is part of the definition of a functor. $F$ doesn't satisfy this, so by definition is not a functor. I've editted in a link to this definition

I disagree. The definition you cite never says F(.) must be a map between F(A) and F(B). Furthermore, consider the dual of my example. In the dual we have f:B-->A which takes negative square roots of negative numbers (in B) to the positive real numbers (in A) - exactly the same operation as F(f):F(A)-->F(B). Yet that is valid in the dual category, so why is it not valid here?

@MPitts Have you read it? I quote: "associates to each morphism $f:X \rightarrow Y$ in $C$ a morphism $F(f): F(X) \rightarrow F(Y) $ in $D$ such that..."

@MPitts And your example in the dual category doesn't really make any sense. In the dual category of Set, the morfisms are no longer functions

"Associates" is not the same as "maps." A map is a function (see Mathstack article 95741). ''Associates" is more general, and need not be a function (thus the frequent use of that word in category theory). Also, by your argument the dual is not a valid category.

A possible solution to my problem that I forgot to mention is that maybe the codomain is not the category Sets, but some other category.

@MPitts yes, $F$ is not a function, since it's domain and codomain aren't sets. However, we are talking about $F(f)$ wich is according to the definition i cited a morfism. The morfisms in Set are functions, so yes, the definition does say that $F(f)$ should be a function and yes, it does say that it should be a function between $F(A)$ and $F(B)$.

The dual (opposite) category of set is a valid category, the morfisms just arent functions, thats not a requirement to be a category

The definition says F(.) "associates" F(A) and F(B). "Associates" does not imply that there is necessarily a function. But of course you are right that in Sets morphisms must be functions. As I mention above, maybe I am in error in assuming F sends Sets to Sets. Maybe F Sets to some other category.

I litterally cited the definition, please read it. $F(.)$ does not associate $F(A)$ and $F(B)$, it associates a morfism $f:A\rightarrow B$ to another morfism $F(f): F(A) \rightarrow F(B)$. You try to define F(f) = f, but f is a morfism from A to B not from -A to -B.

And what other codomains did you have in mind? Sets as objects and diffrent morfisms? Then those morfisms arent funtions so it cannot violate the definition of functions obvioulsy

I don't have any other codomain in mind - that's part of the problem I guess. On the surface F would appear to be a valid functor between Sets and Sets. However, with this example, we see there is a problem. But what if we had not thought of this example? My real question is how do we identify valid from invalid functors? Do you have any insight into that question?

by carefully defining what a functor is. That is what is done in the link a cited, and if you want to check wether a certain map is a functor, you check the definition, it's that simple

Like you, I've read the definition many times. However, we only found the problem in this case by finding the right example. So we might be able to prove something is NOT a valid functor by landing on the right example, but how do we prove it IS a valid functor?

by logic

Of course. But...