Thanks for the interesting answer! There is a reason I want to avoid this, though: AIs see complex but non-AI computers as lacking capacity to consent but still needing protection, like how we view children.

Beyond the objection in the edit, you don't need to worry about how to define or enforce the treaty, you need to worry about how these AI could possibly win a war with such an unrealistic view of the world. They would have to acknowledge that their objections are unreasonable and impractical in order to do so.

I like to remind people that if the Turing Hypothesis is true (that a Turing Machine is capable of representing all possible states of human thought), then a monk moving around index cards according to a rule book is a sufficient computing substrate for intelligence, albeit very slow. Sentience might be everywhere and we wouldn’t notice if it’s “frame rate” (for lack of a better term) is measured in decades.

What does capable of consent mean? You can write a 3-line program that gives (or refuses) consent prior to running the given task. I am sure you can get current LLMs to say things like "yes, I consent to answering your query". Are they truly capable of consent?

@usul Probably able to pass a Turing test, and then able to give informed consent.

(My previous comment is a response to the argument that it would be a "slippery slope" for governing AI to forbid humans from having computers powerful enough to run an AI)

@Stef As long as there's enough data storage space, any computer can run an AI... even a human monk with a whole stack of filing cards and a list of instructions. Do the AIs own the monk too? The only difference between the monk and the most powerful computer in existence is speed.

@MontyWild By the same logic, guns shouldn't be regulated, because any piece of rubber can be used as a slingshot. "The only difference between a gun and a slingshot is speed." And bladed weapons shouldn't be regulated either, because any kitchen knife can be used as a bladed weapon. And yet... bladed weapons and guns are both heavily regulated in every country around the world. Drawing a hard non-blurry line is always difficult. Laws are complex. And yet laws still exist.

@MontyWild Curcumstances matter. If you walk into a park on a sunny sunday with a backpack containing a large knife, a large piece of bread and some cheese, you won't be arrested, even if a cop searches you. But if you walk into a train station with that same knife in your pocket, and a cop searches you, you will probably be arrested and charged for carrying a weapon. Likewise if you have an engineering workshop which contains all the materials for building a gun, you probably won't be arrested... Unless you're also in possession of blueprints explaining how to build a gun.Circumstances matter

@MontyWild So there are hard facts, and there are circumstances, and there is intent. There are things that are clearly legal, things that are clearly illegal, and things that are a bit close to the limit and that end up in court with lawyers building a case containing all the circumstantial evidence.

And that can actually be a really cool plotpoint. Maybe at some point, humans are extremely scare of getting into a war with the AIs, so the human government hits hard on anything that looks remotely like a "complex computer" and close to the grey zone of the treaty. Later humans start being less scared and are annoyed at their previous government, so the new government and cops and prosecutors decide to be very lax and allow anything that isn't too obviously running an AI. And then there start to be tensions with the AIs and risks of a new war if the AIs think humans have breached the treaty.

@MontyWild Thanks for the answer - it's given me a lot to think about - but there's still a misunderstanding. It's not that the AIs want to ban/own computers which could run an AI, it's that the AIs believe programs simpler than themselves still have natural rights (eg, the right to not be used). But, because these simpler programs are not sentient enough to give consent, they cannot ethically be used. The question is, at what point does a system become so simple that it does not have the natural right to not be used? I find the other answers on this post answer this more directly.

This answer seems based around the idea that human rights are founded in property rights. You own your house, you own your body, etc.. This idea is popular among self-described libertarians, but it's not mainstream. Even today, not everyone agrees that property rights are even rights.

@isloe That's a whole other kettle of fish... that your question didn't appear to state. Software sentient rights could indeed be more problematic.

@isloe I've addressed software complexity for you.

@Stef My latest edit addresses your 'is it a gun?' issue.

Well detailed point, but it has a flaw. We humans don't only consider the property of the substrate, but the substrate itself needs to be handled with special care according to our own laws. If we take your position that the owner of the substrate has rights to do whatever it wants, it means the AI can own human bodies and do whatever experiment they want with it. We humans would obviously see that as horrible, so AIs can see our handling of sufficiently powerful computers as such.

@MontyWild You're right - it didn't occur to me that regulating software rather than hardware is the core of the issue. I've edited my post again to reflect this. Your idea of output feeding back into input as a mark of complexity is also very useful. I'm using a mix of answers in my final solution, hence why I'm not accepting any single answer, but thank you for all of your help!

@MickaëlC.Guimarães At the present, human consciousness is inextricably bound to the biological matter that makes up their bodies. Software has no such limitation, by design.