For this question, consider the stable diffusion model.
For a given text embedding, Stable Diffusion can generate diverse images. In this context, 'diversity' refers to the variation among the images generated, meaning that several images with the same semantic content (given by the text embeddin...
@NeilSlater So, are you saying that the 'nosie vector' is the only input that leads to diversity in images generated? Then where does the diversity emerge? Is it in some other hyperparatemeters? If yes,a re they related to neural network or some other?
Are you asking about parameters other than the text prompt. E.g. choice of sampler, cfg scale etc? Or can those be considered fixed, like the prompt. Also are you asking about gamut of possible outputs related to what the different noise values produce, or about the image varying at all?
@NeilSlater I am investigating the factors that contribute to image diversity in the Stable Diffusion model beyond the random noise vector, assuming other settings like the sampler are fixed. My focus is on understanding which additional inputs or some other model related values can influence the diversity of the generated images from the same text embedding under a constant configuration.
I'm still not understanding what you are getting at with "diversity". Do you mean just that the image is different - i.e. the pixels are different colours, or are you referring to the limits of what differences can be realised (e.g. if you ran the same settings with random noise a million times with the prompt "cat", would one of the cats be surfing, or doing something else unusual that typically you would need to prompt for)?
@NeilSlater Diversity here is generation from all the possible images that can be obtained for a fixed text embedding in constant configuration. As you mentioned 'gamut of possible outputs' in fixed configuration.
Still not quite there. Are you interested in that it changes at all, or in the practical limits of what those changes can be? Are you asking if there are other factors beyond noise and settings. Or are you asking "can a cat prompt generate a dog sometimes"?
@NeilSlater I am interested in those factors, like random noise, that help in generating different images from a fixed text embedding in a fixed configuration.
OK. Got there. There are none. You can fix the seed and see this, the same image is generated every time (there are some caveats, because different GPUs may behave differently). If that's not what you mean, please explain?
It is generating different images when I run several times, so I am thinking whether random noise is the only contributor for that since all remaining are fixed I think @NeilSlater
It is possible to inject more noise vectors. Stable DIffusion is open source. If you used something like ComfyUI, or scripts, then you could add noise to almost any of the stages
Yes, pretty sure if you set the seed, it should be the same image each time. Although not familiar with where/how it would be set in that code.
It's pretty common to set a seed then increment it to generate e.g. 10 images that you can then compare when changing other things such as the prompt or one of teh many parameters, to be able to see more clearly what effect they are having
I was kind of hoping you were asking about the limits/gamut of the model - i.e. just how much variance can you expect given you can vary the initial noise seed. I think that is an interesting question - for which I don't have a great answer.
However, this is more about practical control and expectations on a project?
Whilst "what is the potential gamut of a diffusion model" is more philosophical
Discussion between hanugm and Neil Sl…
Discussion between hanugm and Neil Sl…