I feel like there should be plenty of prior art for "don't do things as fast when something is busy" but alas, I'm not a lawyer willing to fight the battle... or a lawyer.

@jcollum, there are lots of bottlenecks besides CPU usage. Today, CPU usage is the bottleneck. Tomorrow? It could be I/O, or network, or memory. So, you check CPU usage and it returns 10%, but the database is running at 99.2% RAM usage. CPU is not the issue. It will still run slow. Don't tie your metric to a specific performance indicator in the database. You want to target something more general so you don't need to tweak your heuristic because you encounter an unforeseen bottleneck.

And that's where increasing the queue size is essential. You don't want your queue to accommodate nominal usage. You need the queue size to accommodate the surge in traffic so your service can work through those messages at a pace the database can handle. Otherwise, you are just pushing this problem around, not solving it. You need a buffer, and the bigger message queue provides that buffer.

With throttling you get a situation when a 1500 instances of MP are sleeping around DB and do no work. What kind of scalability is this? Just limit the instance count (OP's item 3) and then throttling will be done by DB itself!

@Basilevs, if you have 1500 instances that are not doing any work, then how would 1 instance be able to accomplish anything?

@GregBurghardt At least it won't hog resources doing nothing. In other words, if a bottleneck is a DB, do not scale its clients in an attempt to speed DB up.

I assumed, based on the OP's question, that they would know not to do that. Perhaps I made an assumption that isn't true?

What do you think they would do then, using your advice?

I guess I'm confused why someone would ever scale up the number of instances when each one is running slow.

I imagine it is possible to make MP report any kind of DB delays to the cluster controller to prevent spawning more MPs, but it was not obvious to me from your answer.

Just for clarity, MP = Message Process, right?

@GregBurghardt Yes

I'm a little unsure how to clarify my answer, then. Why would you report DB delays to the cluster controller? I haven't done a lot of configuration work with these, so this might simply be a blindspot in my skillset.

I admit I've never saw an architecture that detects when a component is running slow because some of its dependencies are slow. How is it usually done?

The general idea is to slow down message processing. Adding instances of the message processor isn't going to slow down message processing.

How would cluster manager know that it is DB that is slow and not MP internal processing?

I always thought these systems scaled horizontally when incoming traffic ramped up. This question implies the incoming traffic has ramped up, but the downstream dependencies are running slow.

This might be highly dependent on the system used to manage message processors.

Yes, and your answer is to create more MPs. Which is wrong.

Add something to clarify, why cluster manager should not do that.

My answer has nothing to do with creating more message processors. It has to do with slowing down the existing processors.

But horizontal scaling implies that if MPs slow down like that, more MPs are needed

I'm not sure what the criteria is for these systems to determine when to spawn a new instance.

Well, it does not really care about speed, just the queue size, but that's equivalent in this case.

It is probably configurable to a certain extent.

Ok, I think I see what you are aiming at.

I added a paragraph towards the beginning of my answer. I made an assumption that wasn't true. I assumed people would know not to just scale horizontally during times of slowness.

But that paragraph makes all the rest of the answer impractical - the artificial throttling is not needed, because when limited count of MPs hang synchronously in DB write, it causes a natural delay, making artificial delay redundant.

I think I missed some parts of update. I will reread again.

So overall this answer does not help.