Hi. Is a file based tracking system usually good?

The situation is that I have an app that receives email...and there are various steps to process it. Each step is handled in a separate thread...

So there is sort of an in-memory queue.

Each queue has its own folder. This is the way I designed it so that I can track how many items are "added" to the queue...

So when an item is added to the queue, I create a temporary file whose filename is a unique id to the item being processed...

So this is the requirement...is using a file-based tracking a good/bad idea?

it's a lot of disk usage

and that doesn't scale well

of course there would be file proper file cleanup in each queue...

is it still bad?

@deostroll A file system is essentially a kind of database. But is it a suitable database for your use case? Any synchronization issues can generally be worked around without too much effort, but directories aren't very good as job queues: files don't have any inherent order.

You'd have to generate sequential IDs when adding a file and look up the lowest ID when consuming a file. This either involves a lockfile that tracks all IDs, or you have to scan the whole directory each time. I'd rather use a real database.

@deostroll The only way to know if it's good or bad is to first define what "good" means. What characteristics are you looking for? Speed of execution? Simplicity of design? For any non-trivial design you're almost certainly going to want a database, but there's more going on here than just database operations. What you're describing is a workflow.

Hello fellow SE junkies. Does anyone know if there are off-the-shelf products that offer something like SE chat? I really like the SE chat feature set and UX. Nothing else I've used even comes close.

well, that's an exaggeration. Mattermost comes close, but not close enough.

@MetaFight Slack desktop client (I haven't tried it in the browser though)