I would expect asking ChatGPT is not likely to be an effective way to find duplicates. If you want to use AI to find duplicates, the obvious way would be to use something similar to RAG, i.e., use embeddings and nearest-neighbor search. In addition, just saying "it's probably a duplicate" isn't useful if you can't identify the question that it duplicates, which is another reason why ChatGPT is probably not the right tool -- but RAG / embeddings could be.

I suspect a more useful test would be: if you give ChatGPT an old question and a new question, and ask if the new question is a duplicate of the old one, how accurate is it? Does it become more accurate if you provide a detailed prompt that explains when something should or shouldn't be considered a duplicate? Basically, I can imagine an architecture where one uses RAG / embeddings to find candidate duplicate targets, then possibly use AI to check each one to see if it appears to be a duplicate, then use manual review if the results of the AI indicate a possible duplicate.

Question 1 and its supposed duplicate are asking about similar topics, but if you read what they're asking they clearly aren't the same question at all. I think this illustrates very well the problem with this whole "I've decided this is a duplicate on the basis of vibes, now let's find something to close it as a duplicate of" approach.

@N.Virgo This is a common misunderstanding. A is closed as a duplicate of B if A has an answer over at B. It is not required that A and B ask the same. Now please check again the answers, they explicitly address the question of when a result "is probably true". For this reason the close cards are also saying exactly this "has already an answer here...". I also don't find your comment at Q1 fair since it once again misinterpreted my idea.

@MartinBrandenburg could you link to the policy on that please? My understanding is that to be a duplicate, it has to be the same question and have a satisfactory answer, not either or.

And again, I don't think I'm misinterpreting your idea. You want AI to flag questions as possible duplicates based on your intuition that it can pick up a general vibe of duplicateyness from the question text alone, without looking for duplicates. Then you have the idea, which I think is mistaken, that humans would carefully check those flags before acting on them. That you want to do this is stated very explicitly in your question and it's also literally what you put into your test prompt in this answer.

It's not easy to search meta so I could be wrong, but it seems like this is the closest thing this site has to an agreed policy on duplicates. While it's not completely clear how to apply it to this example (question 1 above) because it's aimed at more precise questions, it's pretty clear that it does matter how closely the questions are related, not just the answers.

See also official Stack Exchange blog posts here, here where the issue is discussed, in which it's argued that even differently worded versions of the same question shouldn't always be considered duplicates, if they're likely to be found by people approaching the question in different ways.

@N.Virgo That's a poor argument for rampant duplication, Instead, if the dupe contains keywords or phrases aiding search then simply ensure they occur in the canonical dupe target. Generally, when composing Q&Q's you should check that they can be easily located by simple obvious searches.

@BillDubuque nobody's arguing for "rampant duplication". The claim is that the question "what do mathematicians mean when they say a result is probably true?" is a duplicate of a question asking [paraphrased] "does there exist a formal method of proof where you give a carefully chosen list of examples and that suffices to prove the theorem?" The answers to the latter cover some of the ground asked for in the former, but should that by itself suffice to consider them duplicates? The links I posted suggest not.

@N.Virgo My comment said nothing about that specific question. Rather, the point is that the generic SE recommendation easily leads to rampant duplication on this site. That's why I proposed abstract duplicate closure 14 years ago. Time has proven that it works quite effectively here.

@BillDubuque oh, that's great - that's the policy I felt I was probably missing when I said it's not easy to search meta, just above. Then the question is whether Martin Brandenberg's suggestion of using AI to flag "suspected duplicates" without actually checking for duplicates would be effective in implementing that specific policy. I suspect it wouldn't.

@N.Virgo You keep misreading what I write. I didn't speak of flags. When a question is likely a duplicate (or of low quality, we can use the same idea here!), my suggestion is that it should not be immediately published. There is no flag on a published post like the current ones. It just goes into a queue that needs be checked and approved for publication then by other users. If you have other ideas to prevent obvious duplicates being posted which usually don't get a moderation afterwards, please post a separate answer.

@downvoters: What is wrong with my answer here? How can I improve it? If you dislike the idea of using AI, then this is not a reason to downvote my answer. My answer is an attempt of an analysis of this suggestion. You should only downvote if the analysis is not correct.

I do not understand how "it's not a flag" can be squared with "it goes into a queue". Those are the same thing, and "it goes into a queue" is exactly the thing I'm arguing against. I'm saying AI could productively be used to help empty the queue of duplicates but that it should not be used to populate the queue in the first place, and also that putting things in a queue of duplicates without actually checking for duplicates is a bad idea regardless of AI. I believe I've been pretty clear why I think so.

I gotta say I think both these points are pretty straightforward. Certainly they're subjective and a person could certainly disagree, but I don't think there's much to discuss about them beyond that. I sort of wish I'd left them as comments under the question because of that. It would probably be more constructive to propose alternatives that I think would work better instead. I'll think about that and post another answer if I come up with anything.

This doesn't really seem at all helpful to me. So what if GPT outputs "the question is likely a duplicate"? This doesn't help anyone to find a duplicate question, nor do I believe that GPT is even a reliable predictor of whether or not a question is a duplicate in the first place. "AI" tools for detecting duplicates will only ever be useful if they can forward potential duplicate candidates.

@N.Virgo Just for the record, my moderation philosophy is "If question [A] has a relevant answer below question [2], then question [A] should be closed as a duplicate of question [2]". The goal of closing a question as a duplicate is not to punish anyone, but to connect the person asking the question with a useful answer, and to provide a signpost to future readers who might land on question [A], but find their answer at question [2]. If someone demonstrates to me that [A] has an answer below [2], even if the questions are phrased differently, I will happily dupe-hammer [A].

@XanderHenderson your last message is contrary both to Stack Exchange guidelines and to agreed policy on this site. Please check the links in the thread above. If your moderation philosophy is contrary to policy you should consider changing it. I agree with your other comment though.

@N.Virgo I am familiar with the links above, and I see nothing contained in the content of those links which contradicts what I stated above. Perhaps you could highlight the places where you think the contradictions exist?

@XanderHenderson well, the SE blog posts both argue against closing questions as duplicates based only on the answers without considering the question text as well. The contradiction between this and what you wrote seems very clear-cut. Additionally, Bill Dubuque's "abstract duplicate" link says that you should consider closing both [A] and [2] as a duplicate of [iii] if [iii] is more general than [A] or [2].

I will heavily change my answer here. I think it is just so easy to misunderstand my point here. Sorry for the comments which are then probably not making much sense anymore.

@N.Virgo You are still not providing any direct quotes from any of those sources. I continue to maintain that if [A] has an answer below [B], and if that answer will resolve the asker of [A]'s question, then [A] is a duplicate. Again, the goal is to connect the asker to an answer. In particular, at stackoverflow.blog/2011/01/05/…, the author suggests "But it's not OK to close it as a duplicate of a twenty-seven page guide to netmasks. That's the moral equivalent of saying 'RTFM.'" You are basically demanding that I RTFM.

Please provide a more precise citation.

@XanderHenderson direct quote from the first blog post: "That's why we actually don't mind having several versions of every question, where there are variations in wording or circumstances."

@XanderHenderson direct quote from the second: "...duplication is not necessarily bad. Quite the contrary -- some duplication is desirable. There’s often benefit to having multiple subtle variants of a question around, as people tend to ask and search using completely different words, and the better our coverage, the better odds people can find the answer they’re looking for."

@XanderHenderson from the second: "it's OK for duplicate questions to have duplicate answers. While you could argue that the duplicate questions could all be merged into one question with a "master" set of answers, this is kind of irritating from the perspective of the user looking for an answer. Put yourself in their shoes. Instead of finding 'Duplicate Question Duplicate Answer' They have to deal with finding 'Duplicate Question [closed as duplicate of Question] click here to see answers'"

@N.Virgo This does not say that those questions should all be open. Again, a closed (as duplicate question) still serves as a signpost. There is a difference between deleting a question, and closing it.

And that completely misses the point of "If [A] has an answer below [B] (i.e. if the asker of [A] had found [B] before writing their question, they wouldn't have asked it), then [A] is a duplicate.

@XanderHenderson I'm not arguing against closing duplicates. I'm arguing against blindly applying your approach of "If question [A] has a relevant answer below question [2], then question [A] should be closed as a duplicate of question [2]". It can simultaneously be true that most duplicate questions should be closed, and also that not all should. Usually [A] should be closed as a duplicate of [2], but sometimes both should be left open and sometimes both should be closed as a duplicate of [iii]

Who said anything about "blindly"?

Your use of "if" suggests you treat it as a rule.

@N.Virgo I'm not a machine---I am capable of using judgement.

Regarding your comment about "moving things to chat where no one can see it" (1) that is nonsense---anyone can click the link and see the chat, and (2) chat allows one to reply to a specific comment. You wrote several comments in a row, and I wanted to reply to specific comments without confusion.

That is what chat is for.

Well ok, but you're replying to a discussion where I was saying (effectively) that if question [A] has a relevant answer below question [2] then some judgment still needs to be taken into account in deciding to close [A] as a duplicate of [2]. Your response was that in that situation, "question [A] should be closed as a duplicate of question [2]." So you can probably see why I thought you meant it always should. If you didn't mean that, then what was the point about moving it.

sorry that last sentence was meant to say "what was the point you were making?"

genuine apologies about that other comment, I shouldn't have posted it and deleted it immediately afterwards

@N.Virgo Comments have limited space, and nuance is hard to convey---every moderation decision requires judgement.

But, generally speaking, I see the proliferation of duplicates on Math SE as a much larger problem than the incorrect closure of questions as duplicates. In general, if I believe that the asker of question [A] would have opted not to post that question if they had just Google'd first and found question [B], then I have no problem closing [A] as a duplicate of [B]. Again, the goal is to connect answers to questions.

@Xander Please, please do not demote meta comment discussion to chat. One of the primary purposes of meta is to support discussion. I strongly* disagree* with this being moved to chat. It hides discussion of one of the most important issues challneging the site.

@BillDubuque It is not a "demotion". Again, chat is better featured for actually having a conversation.

Xander, I agree there for sure. We're only having this discussion because of one of the rare edge cases ("question 1" in the answer under which this was originally a comment thread). That's a case where an answer to question [A] arguably exists under question [2] but I would say judgment should be exercised in not closing [A] as a duplicate of [2].

Note, also, that most of the conversation here has very little to do with the answer---the answer is about using GPT to detect duplicates, while nearly all of the conversation is about how dupe closure should be used.

@N.Virgo There are certainly edge cases, but I don't think that they are worth getting one's knickers in a twist about. Policy should focus on the majority cases, not edge cases. We have real people in the loop to deal with the edge cases.

(on the meta discussion there are some comments here that are relevant to the answer though. I'm happy for all of mine to stay here but some of the others could be moved back to support the discussion there.)

I very strongly disagree that policy should only focus on the common cases.

@N.Virgo You can't write policy for edge cases. There are always going to be exceptions. If a policy covers 90% of cases, then I call that a win.

Yes you can. That's what meta is for.

If a policy fails in 10% of cases, I call that a disaster.

@N.Virgo I very strongly disagree. As soon as you write a policy to deal with one edge case, another edge case becomes a problem. You patch that, and some other edge case pops up.

Yep, that's why we have a system for having meta discussions and archiving them, so that we can build up a suitable body of case law that resolves as many edge cases as possible.

It is not possible to write policy that will cover all the edge cases. So policy is written to cover the vast majority of cases, and human judgement (in the form of moderators, or discussion on meta, or whatever) handles the exceptions. Trying to write policy to cover the edge cases is just asking for people to look for loopholes and to rules lawyer.

I think we will have to agree to disagree on this point, for now. I feel that philosophy doesn't lead to the best outcomes for the site but I don't think I can convince you of that. Perhaps I will post about it on meta but I don't think I'll have the energy any time soon.

btw I'm closing this tab now and I don't know how to get notified when there are new comments, so I guess that's so long from me. (So much for chat being better for discussions.)

@N.Virgo You get a notification in your inbox if someone pings you in chat, same as if they pinged you in a comment.