You say it's an ethical question, but it seems you are asking can an AI intentionally change its core aims, not can an AI morally change its core aims (i.e., is it morally permissible or not).

@Hokon one can think about the original aims as of embedded moral.

No need for any ai. A program that's just a few lines of code can do that. It's so easy that the question is silly, I am afraid.

Also the question supposes there is "source code" that the aims are encoded in the source code, none of which makes much technical sense. Those are unnecessary restrictions, indicating lack of understanding of technology.

@tkruse I meant he has access to any kind of own functionality, be it source code, training data or whatever.

Are you asking whether changing one's aims has a bootstrapping issue? Even thinking of wanting to change means there has already been a change, and, how can it occur to one to change? Complex systems like thought patterns change randomly all the time, so any reasonable idea is bound to arise sooner or later. Otherwise there would be no point in having a mind.

@ScottRowe yes, I am asking about it, and the idea of a random though could be a workaround, but would not an agent with clear goals make efforts to eliminate random thoughts?... Or abide by them?...

Without some amount of "randomness" and "lucky errors" there cannot be creativity. It may be unavoidable (and desirable) that any autonomous intelligent agent will have some random thoughts. Humans also change themselves from the ground up on almost all levels (pacemakers, brain interfaces, psychotropic drugs). Marvin Minsky once wrote: Will robots inherit the earth? They sure will. -- They will be our children.

@mudskipper well, let's assume the agent can get experience, and in a changing environment, and otherwise completely free, including in decisions that can introduce randomness (to the extent he thinks it is desirable), in creating other AIs and clones, new models, as long as he thinks this benefits his stated aims.

I guess the question is: how do agents come up with new ideas? Since they do, and because it improves survival, it must be a good idea. "Even if you're on the right track, you'll get run over if you just stand there."

@Anixx - I don't have a direct answer to your question -- I believe it's too ambiguous. But it seems similar to a question that I've been wondering about: Can it ever be rational to make a decision that one knows one will later regret? It seems that CDT (causal decision theory) in some of its variants is answering this with "yes" -- which seems rather weird...

@ScottRowe no. An AI can come with new ideas without changing its own goals. But how can he come with an idea to change the goals?

@mudskipper quite similar, yes, but this is a reverse. Can one ever come with a decision that is contrary to the current goals, anticipating that the decision will change the goals and as such will not be regretted?

@Anixx - The change of its goals -- even if those are totally hardcore built-in -- may be a side-effect of implementing its new idea. That's the beauty of being irrational and inconsistent: we never completely know what we're doing. (How else would we become addicted in the first place? Etc.)

@mudskipper yes, but the agent may anticipate this and so, refuse the idea because it hampers the original goals. Or not? This is a question.

@mudskipper Would an agent decide to manually change the goals if the changing environment made the goals inconsistent, meaningless or ambiguous? Or he will stick to them regardless and will try to find meaning and interpretation? Of meaningless goals would incaacitate the agent so that he would not have any desire to fix the things?

I believe both are possible, yes. So, yes, if the agent has enough knowledge/understanding that some action might change its basic goals, it might not start experimenting with hard drugs (for instance). At least, that's my main reason to stay away from them, even though I'm very curious. I just don't want to slide into becoming a junkie and losing all moral values. "Curiosity killed the cat" :) But sb might also be forcefed some drugs. There are scary stories about that...

I actually now also see a connection between your question and Camus' argument in The Myth of Sisyphus. Camus: We don't have any goals. All our goals are ultimately absurd. Suddenly (or for whatever reason) someone becomes aware of that (or comes to believe that). So, what to do then? -- Realizing this (insofar as it's realizable) does change one's way of being, I think :)

@Hokon this question is important for understanding of AI ethics.

You're 3/5 VTC, so I'm proposing the following edits to avoid closure. The question as is has too many questions and insisting it's an ethical question muddies the water. I encourage you to roll back if you reject the proposal.

+1 To @JD comments -- JD is opting for the most charitable interpretation :)

@mudskipper It seems to an incorrigible fact of life that the Two Cultures is a major obstacle to intelligent discourse about the philosophy of computer science. Many of our colleagues refuse to read books, and many philosophers refuse to read source code. I'm just here to bring the two tribes together. ; )

Humans certainly seem able to do so. But like the old joke about how many psychologists it takes to change a lightbulb, they have to want to change.

It's at 4/5. If this gets closed, feel free to post it in Meta and we'll see if we can't keep something open.

If we get rid of the metaphysically loaded language (goals, decisions, intentions, etc) this looks like the alignment problem.

@JD I interpret ethics as everything concerning the rules of behavior, not only answers on the question "should?"

Your question boils down to "Can AI have free will?" How the heck do we know that, when we probably don't have free will?

@RonJohn so, if the AIs cannot modify own aim, they are not dangerous and will always remain aligned?

They might modify their aims if we (possibly inadvertently) programmed them with that ability. But does that mean it changed it's own aim, or does it mean it followed it's programming?

Maybe we should program people to do prosocial things? And not let them change that goal...

@ScottRowe that's the goal of religion, I think (for each religion's ideal of society).