Sorry, but I have an extremely hard time seeing how this is in any way credible. We have a lot of experience with "unthinkable questions", we call them either paradoxical or nonsense. Such a question is about as believable as the Monthy Python's Joke that's so funny it kills everyone that hears it.

@pipe If the Monty Python, or yourself (except if you're an AI), can think of this question, then it is not unthinkable for humans. The point is this: the question is unthinkable, you can't imagine it, nobody but the IA can.

Monthy Python couldn't think of the joke (or question), they just presented the idea that such a joke could exist, in the same way you present the idea that such a question could exist. It's that idea itself that I don't find credible - the human mind just ignores things like that.

Also, it's not totally unplausible. Read Dawkins "Viruses of the mind". Also, you know this song that you can't get out of your head? Or these people that can't stop thinking "What I'm gonna do? I'm stuck in this horrible situation, lost job, ..." until they commit suicide, even though, from an outer perspective, the problem seems not so big. They are stuck with the question and can't focus on searching an answer. The mind is strange, don't underestimate its strangeness and the amount of things we don't know about it.

@Piskvor no, because the AI is clever. It would not generate a random serie of words and considerate it a valid question. The question makes sense to this machine, and to the humans. It was just not possible for the humans to think of it.

That's an interesting proposition, but one that's hard to think of - can an unthinkable question even make sense? (OMG, it has already begun! ;)) More specifically, is this enough to keep the readers' attention, or will they shed the load by reducing to "so there's this lethal virus, except this time it's a meme"?

@Piskvor :) you're lost, now you will only think of this. About the interest of this, well there are books in which nothing happens and that are considered masterpieces. Also, I'm not trying to find a good story, I'm trying to find a solution to the question the OP asked.

Aha, it's en.wikipedia.org/wiki/The_Zahir then - I knew it reminded me of something. I do concede that your points are valid and the answer is applicable :)

Hofsteader’s “this record cannot be played on this record player” theory (see Pulitzer Prize winning non-fiction “Godel, Escher, Bach”. Also similar to a Twilight Zone episode I saw once about the ultimate joke: once heard, humans could not stop laughing.

Also, Asimov’s short story “Breeds There A Man” where mike scientists keep committing suicide after a particular discovery. Real world: extreme rate of suicide among researchers of infinity and cardinality in mathematics. And the problem of studying nihilism and existentialism.

How could an engineer spot this difficulty coming up?

Another story that makes use of this idea of an unthinkable question: The Riddle of the Universe and its solution, (c) 1978 Christopher Cherniak

@RossPresser just as I did, by discovering the three rules and deducing what will happen.

@Legisey Then how about if we pose to the computer, "Please find us a way to avoid this situation."

@RossPresser Answer: keep asking questions! (Which is exactly what OP means, I guess, when he talks about 1001 nights)

"Keep asking questions" probably wouldn't work though, as eventually all the questions would have been asked, and when that happens the AI wouldn't need to analyze the question to come up with the correct answer, but instead just retrieve the answer from memory, so the humans would need to ask questions exponentially faster or else come up with new questions, which may be impossible for this situation to arise in the first place

@BladeWraith That assumes that humanity can only conceive of a finite number of questions.

People, are (un?) fortunately fundamentally irrational creatures adept at self-denial. This killer question will die at the hands of ignoring/misinterpreting said question. You'll get a lot of "That's an interesting question" and then everyone just continues with their lives. Now if the question starts an ideological war instead...

So the computer becomes Cthulhu?

@called2voyage in theory there is a finite number of ways any and all languages can be constructed together to formulate a coherent sentence, simply asking the AI "Seven Sausage Kill Multidimensional Wobble" while it contains Verb, adjective and noun, could only produce basically a syntax error that would be discarded, next question, therefore there is a limit, and within that limit are a multitude of questions that ask the same question worded differently therefore same answer from memory

@BladeWraith Yes, but language changes over time. New concepts demand new words. If we were trying to ask all possible questions in an instant, the set would be theoretically finite, but given an eternity the language will continue to morph and new questions will become available.

@called2voyage this could very quickly become a thread of its own... i'm genuinely not sure whether the evolution of conherent language would grow faster or slower than the speed at which you could ask the questions... perhaps simplify it and have the AI not be able to grow its language library beoynd a certain point and the evolution of language could cause the problem in the first place, perhaps the computer cannot answer a question that does not start with What why where when, how or if, language has evolved to the point those words are no longer used...

@BladeWraith It's easy to come up with infinite questions: "What will happen in 1 year?" "What will happen in 2 years?" "What will happen in 3 years?" And so on. Or, if you prefer recursion, the question, "Did you know that I know that you know that I know my name?" can be arbitrarily extended.

@BrianMcCutchon Yes, but from a certain point, the answer will always be « I have no idea ». So the AI can develop a simple rule and consider all these new questions as being the same one. If year > very big number -> answer = “It is not foreseeable”. Very interesting point though.

@BrianMcCutchon yes the human ability to ask questions can be extended ludicrously, my original point is when does the finite words in the English (or any) language reach the stage where every possible combination of words that are capable of forming an "independent" coherent sentence, without the answer being drawn from memory due to a previous asked but simply differently worded question, and also we would need to ask questions faster than the AI can answer them in order to advert the crisis, huge extended questions like that reduce that ability to do so