Erik Eidt

possibly relevant: Why Pascal is Not My Favorite Programming Language, Brian Kernighan

a*b+c*d doesn't need the user to declare an explicit temporary variable either; language implementations make their own internal temporaries as needed.

The job of the compiler and assembly language programmer are different. The compiler translates a well understood language like C or fortran into assembly language. The assembly language programmer is both writing the algorithm and the assembly code at the same time, rather than translating pseudo code or C code. So, these are not really equivalent operations.

VMS is good for Cobol-style record-oriented flat file databases. These databases have the file system as their interface. Multiple applications can be accessing the database files at the same time, hence record locking etc.. are desired. These days, however, we have moved to database applications as a service (e.g. SQL) rather than using the file system as the common access service. Thus, what we need more and more is for the OS to get out of the way while enabling applications each to do highly custom algorithms and data structures, including using their own file structures and formats.

No quotes or text from Culter, though lots of critique of unix: The Unix Haters Handbook

@Chris, if ever manually edit generated content, that edited content becomes source code, and you should have it in git, and assuming you want to keep the edits, you should also protect against regenerating those edited files b/c you'll loose those edits (which is why it becomes source once you edit them.)

@Chris, i'm not sure it matters that much to your daily life. however, the word source in source code means original, as in some originally authored source. it doesn't mean text or human readable: that's an orthogonal property. A binary image file can be source if it is authored by a human (especially if it can be edited). anything generated is just not source. so a thumbnail is not source since it is generated from an image (unless you loose the original image...). you can still check generated material into your git, though, no problem, whether human readable or binaries.

@Chris, I would disagree that code feed into a tool is "source code". It only source code if humans edit it. For example, some compilers generate assembly code, which is feed to an assembler. That doesn't make the assembly code source code even though it is text. However, such a project may also have human generated assembly code, and that is indeed source code. Consider what happens if you loose some code? Source code means it would take an engineer and time to reproduce it, whereas T4 generated C# would be instantly reproducible from the original template (which is source).

Yes, one reason we generate human readable text (not source, but still code) from other (true) sources, is to make up for shortcoming in programming language abstraction and construct parameterization. (Another reason is because we are using a multi-stage step-down in the programming language level --- some very high level source language translator generates C, meaning we can leverage the C compilers to generate binary across platforms.)

Our languages do not support paramerization everywhere and anywhere. Generating code can be used as a method of abstraction, in the sense that we can abstract common functionality into a reusable source unit, reusable via some parameterization, which may go to generating different forms of the same. Usually this means that the generated (text) code is repetitive, but the original is not (it is DRY by comparison). It is an effective technique, when other parameterization mechanisms (e.g. the arguably limited ones built into the language) fail to allow for the commonality in repetitive code.

@Utku, its just one way of doing that. One easy way with high leverage. An easy way to interface with (original) source code in the programming language.

@whatsisname, ok, but let's not suggest that harvard architecture code is not first manipulated as data by the development environment.

What I'm proposing should attempt all matches. Only the gatherer announcing that individuals have been matched would halt matching for them.

Effectively, each worker is maintaining their own copy of the data, and of course, would only concentrate on their portion of the problem, so then, not really a full copy of the data, only their subdivision.

Still, if in-memory doesn't work, each worker could use a local disk-based persistence as well.

If you need more workers you can add lots of them, and they'll each only store their share of unmatched elements in their local memory. However, you will have to decide if an all memory method will work for your domain and expected load.

This is why I specified that the gatherer would broadcast the retired elements back to the workers, and, one direction messages (e.g. no reply required) can scale very well, since no one is really waiting on a reply.

I was suggesting that the workers could maintain what/who they're trying to match in their local memory, and when they receive the broadcast from the balancer that there are new candidates, they would each do their portion of the matching then from memory.

All makes sense.

let's see if we can take this to chat

I was thinking along the lines of using simple messaging as in (micro) services and putting any necessary persistence more or less on the side.

The gathering might scale out if you establish predetermined rules for which match to accept in cases of multiple matches found, such as lowest IDs. This instead of relying on choosing the first to arrive by messaging. It would might take longer to inform the workers of elements that have been dismissed but at least the workers can be scaled further as needed.

How to improve it? One common option is to use a sorted or balanced tree. This improves the performance of find (which appears to be in use by all other operations, e.g. find, delete). The other way is to provide a parallel data structure that is specifically designed around mapping from IP address to some node location/reference in the tree. Then encapsulate both parallel data structures (the map, and the tree) into a single larger data structure.

While the actual delete would in theory be improved by the doubly linked list (under certain circumstances), the information it provides can be recovered for free during the find operation.

Rep #3 does not improve on Rep #2, for delete, because: the doubly linked list does not help. The reason it does not help is that the delete IP Address operation is always proceeded by a find IP Address in the tree operation, and the find operation is not improved by the doubly linked list.

cheers for now...

sure, assuming you find this potentially helpful.

sure. I would keep the existing tree and merely augment with the look aside dictionary as separate data structure. oh, sorry, i have to run to a meeting. let's take up later if possible.

what are you thinking?

always ;)

so the dictionary just gives you fast search into the tree, but is not the db.

I'd probably be thinking IP address is key and tree Node* is value.

(that lookaside would be the dictionary)

ok, so you can maintain a look aside to find the element in question faster.

do you search the root, then all its children and so on?

right, so how do you perform the find operation?

(the item* associated with the IP address)

ok. what do you do to find the IP address in the rooted tree?

ok, so when we say insert or delete, we don't pass a rooted tree as parameter, because it is understood which rooted tree, right?

um, what I'm asking is if there is only one rooted tree per router?

and, is there pretty much one overall tree per machine?

does insert also have an IP address?

ok

so this delete by a given IP address should delete multiple things?

ok, and how do you identify what tree?

ok, so what does delete API look like then?

Ok let me make a more general comment, for a more general situation. Let me back up for a second to discuss the encapsulation breakage. Let's consider a simple singly linked list. Now, this is not a set, so the same item can be in the list twice. How do you tell the data structure you want to delete just one, and which of those two items? You have to provide some indication of which one. This can be done with good encapsulation based on the methods chosen to be exposed.

Now that you have the tree node, you can do a delete on it. Wrap all of that in a separate class and you have encapsulation.

Yes, depending on the API involved. Like I said, if you have to work with the nic and not the tree node, then I'd have a dictionary that takes NIC in as key and returns the treeNode as the value. This would be internal so as not to let the tree node escape the local context.

However, let's go on a brief tangent...