Why does the container world mean you can't talk to any message queue you damn well please?

You said you can't use a local message queue because "in a container world is that there's no such thing as a local anything". If I can talk to any message queue why is a local message queue a problem?

@CandiedOrange as far as I can tell, by that logic I don't need a local message queue, because I can talk to any message queue. The point I was trying to make was that there's no in-process queue functionality, so given normal container practice the "local" queue would have to be another container, just as likely to be restarted as the "main" queue would be, and thus no more resilient. The idea of putting a "local" install in seems to be more appropriate to VM/real machine thinking, where multiple systems can run together, with no fear of one node going down or a network split.

I still fail to see why containers complicate this problem. You either have the rights to restart a message queue process when it goes down or you don't. I'd failover to a fixed amount of local memory before shoving it to disk. But I think it'd be better to just have a backup message queue running to fail over to.

This isn't anything to do with having the rights to restart a message queue. It's "what happens if the MQ goes down, and then the message producer goes down". Ideally when they restart, messages should have not been lost. In old-school ESB land, there'd probably be a box to tick to locally queue messages on the ESB. With, say, Pika, messages are held purely in memory and can be lost. A backup queue is a possibility (we've got a cluster of 3 rabbits), but that feels a bit like adding another hull onto the Titanic, rather than addressing the problem itself.

You're "problem itself" is not being communicated effectively. I'm challenging you to find out exactly what it is. I still don't see what containers are doing to get in your way or how local helps you other then avoiding speed of light delays. Writing to disk has very similar if not larger delays.

I'm not worried about delays - I haven't mentioned delays, rights to restart queues, or not being able to talk to something because it's not local.

Well that just shows you haven't communicated the problem. I'm left here trying to imagine what it is.

That's one possibility.

Message producer P tries to send a message -> Rabbit R is down -> P restarts. What happens to the message?

^ that's the context

failover succeeds or fails. You can't perfectly fix a problem where things are dieing. The only way to guarantee transmision would be a to resort to a TCP like relationship upstream where you resend until receiving an ack. You can't blow up everything and expect it to work.

"The only way to guarantee transmision would be a to resort to a TCP like relationship upstream where you resend until receiving an ack." - how would this work if the producer restarted? Where would it get the message from?

Like I said, upstream. If there is nothing upstream you're simply asking the impossible. Nothing works when it's dead.

I'm not saying it's dead, just that it's restarting, or needing to be restarted

What do you mean by upstream? The producer?

events take time to propagate. Shoot what's moving it in the head and it dies. If nothing else knows it existed it's gone forever.

upstream would be whatever held the event before it got here. If that exists it can try to deal with the failure if it's smart enough.

So why can't the producer be smart enough to do the same? What in your mind makes one capable of that and not the other?

Or more on-topic: why do you think the option proposed in the question doesn't solve this?

It can if it isn't what you're shooting in the head

Well yeah, then I may as well get rid of the producer if there's a magical producer upstream that never goes down :)

Your question doesn't make any of this clear. You don't get a magical producer but you could make clear the situation you're in and what exactly you're trying to mitigate.

Doesn't make any of what clear? I'm responding to what you're saying

The situation is still the following:

Or as per the question: "The problem: what if RabbitMQ is unavailable to receive a message publish (e.g. network down; RabbitMQ down) and then the publishing system dies/restarts? "

And if it's not clear from "What happens to the message?", I don't want to lose the message.

You lose the message if the only thing that knows of the message dies before communicating it to anything else. The problem is the question is still raising an issue about containers and locality that is really not part of this issue.

@psr, maybe you can explain what the question needs better.

Yeah I know that it loses it - that's the context. I'm really asking for gotchas on the proposed solution and or whether there are standard ways of solving this problem in a robust way.

It's your question. Try to save it before it dies or don't.

Honestly no idea what you're missing from the question

Then you haven't been paying attention. Your question has an edit link for a reason. You haven't touched it once. Sorry but I've done all I can for you.

I'm happy to edit the question if I knew what to clarify

What's unclear about "Message producer P tries to send a message -> Rabbit R is down -> P restarts."?

And I know you've said it's impossible to keep the message if P and R are both down, but what about the suggested solution (on message publish to write the message to disk as a new file, and have an async process monitor the folder being written to and publish the message (and then remove the file). I.e. a store and forward pattern.) doesn't work ?