@NeilSlater, learning statistically by example is correct. The use of the Jacobian or Hessian matrices to generate a corrective signal is correct. Distribution of that corrective feedback to the attenuation matrices in sequence opposite to that of the primary signal path is correct. The application of probability and statistics is rough in a few places but not grossly. The mathematical imperfection proposed here is the additive correction of parameters, an assumption around which the other math is oriented, and, I believe, an problematic one. I explained further above.

@NeilSlater, you misunderstand what I mean by distribute. If the entire corrective signal were applied to each layer when converging to a surface with low curvature, the correction would overshoot by a factor approximately equal to the number of layers, which is why the learning rates are generally set below the reciprocal of the layer count (α <= 1 / J, where J is the number of layers). Although the corrective signal is multiplied by the partials of the activation as it is propagated backward, the effect of the layer by layer process is the distribution of the correction.

@DuttaA, When you wrote, "We need to have a total analytical understanding of [nets,] which I am not sure if anyone has," you zeroed right in to the crux. Additive updating was simply the easiest to get working first, but, for the reasons I gave, perhaps not very indicative of a comprehensive analytical understanding of networks.

@NeilSlater, The question of whether to use the 3rd Taylor Series term (use the Hessian in addition to the Jacobian) is interesting and perhaps not sufficiently well covered in the literature . The contention I have is downstream from that computationally and upstream from that conceptually. Although I agree that it would take 80,000 words for me or you or anybody to write a good answer now, I'm guessing it will only take 80 words for any of us to answer it later. We will then understand what kind of assumption was behind the decision to adjust the previous parameter value via mere addition.

I am quite familiar with Neural Networks, but have literally no idea what an "attenuation matrix" is, or what the attenuation factor p refers to. Having p = p + ... as an "equation" in there also doesn't really have any mathematical meaning at all, the single most important part of that equation has been replaced with three dots. The second "shortcoming" is just a single sentence, stated as a fact, but I think that sentence needs a whole lot more elaboration. Based on this, I agree with @JohnDoucette that the terminology in the question makes it difficult (impossible) to understand.

@DennisSoemers, thank you for asking for some clarifications. An attenuation matrix is a matrix P that attenuates a vector A from functional result of layer i before feeding the multiplicative results into the functional input of layer i + 1. ... some of the literature uses w sub i,j for the elements but the entire set of w is the parameter matrix being tuned at that i to i+1 layer interface. I assumed that was made clear when I said, "The additive parameter adjustment p = p + ...," but I was wrong to assume that, which is why I clarified in the math under, "Responses to Comments."

@DennisSoemers, the second shortcoming is indeed a fact both theoretically and practically. It is well known, so one wouldn't think to explain it. Especially since the difference between local minima/maxima and the global minimum and maximum of functions is college board exam material. If it is not taught in high school, it becomes a first year college requirement for any undergraduate science or technology degree program. One cannot understand convergence or back propagation at all without at least that first year calculus and analytic geometry background.

@DouglasDaseeco Right, for the matrix, I was already suspecting that's what you meant, but I've literally never ever heard anyone refer to that as an attenuation matrix. Again, nothing shows up on google either. I'd love to see if you have links to anyone ever actually calling it that, but as far as I'm aware, the rest of the world simply calls that a "Weight matrix" or "parameters". Maybe the same thing is going on with that second shortcoming, maybe it's something that indeed is obvious once phrased using "normal" words, but right now it's not obvious to me (and I suspect few others)

@DennisSoemers, I knew you would understand attenuation, if from memory then from context. The current craze in ML has produced a body of people who have taken neither calculus nor signal theory. I use terms like vector-matrix multiplication and attenuation and push the true meaning of topology, which many are taking to mean the number of elements in each network layer, absolutely incorrectly. It is not to confuse, posture, or self-promote. This body of interested parties need to learn how to read and converse in well-constructed mathematical and engineering language. I do the same in our lab.

@DennisSoemers, The second shortcoming is, like I said, first year calculus. When you determine the derivative of f(x) and set f'(x) = 0, you get the critical points of f(x), minima, maxima, points of inflection, or saddle points in higher dimensions than 2. Among the minima and maxima, there are extremes in the local region of the curve (or surface, in higher dimensions than 2) that are not the minimum or the maximum of f(x) over the entire region. In classical iterative searches for the global minimum or maximum, an algorithm can inadvertently find a local minima or maxima.

@DennisSoemers, As an artificial network learns, the algorithm employed seeks the error function's global minimum in the error function. If the algorithm finds a local minima that is not the global minimum, it will detect convergence and cease iterating. For this reason, the literature suggests (a) Attempting to use info to initialize close to the expected global minimum, (b) Injecting pseudo-random noise, (c) Runing from more than one starting point.

@DouglasDaseeco Yes I agree that is a relatively obvious fact. But that's the second of what you named three "negative outcomes". The thing I was talking about is the second "shortcoming", which is: "The trend of the disparity between actual and desired network output is not correlated to that of the parameter matrix." That is the statement that does not seem obvious to me at all.

I understand, and you are correct. The first bulleted list is a list of two shortcomings and the second is a list of three outcomes. I'll explain the second from the first list, which may become clearer when you focus on the word "trend".

In the neural net of the brain, the neurons have organelles. There is memory in them. It is for this reason that they may (although no one has proved it yet) be Turing complete. ANNs are not because they have no memory. The adjustments to attenuation, unlike that of mammalian neurons, are stateless at the layer level and cannot apply trend. The use of the Hessian, the work in RNNs, and the more recent Attention network research attempt to rectify this shortcoming.

From the neuro-biology and genetic end, as well as traditional cognitive science, researchers are approaching from other directions.

right, so basically your concern is the lack of second-order effects / momentum / memory (roughly)?

There may be 1,000 problems current ANN design. But this question focuses on just one of them: The fact that the adjustment signal (see formula in the Responses to Comments section) is added to the parameter value. At the zero side of the parameter range a delta of 0.1 is a major deal. Such opens a connection that did not exist at all before adjustment. At if the parameter was 0.1 below the max for the parameter range, a delta of 0.1 means much less.

That means that statistically, the adjustment signal is skewed, which is not optimal from an information theory perspective.

That aside for the moment ...

... returning to the second shortcoming ...

... second order isn't a correct term (even though I know people are using it for this trend / memory idea).

The correct term is temporal.

In Turing's machine, he attempts (successfully) to define a small mathematical domain that was complete (in response to the crisis Kurt Gödel created when he proved that a formal mathematical system could not be guaranteed complete. He proved that there were truths that could not be proven.

One of the features of Turing's imaginary machine was that it had a temporal element, which we now call memory.

By, "The trend of the disparity between actual and desired network output is not correlated to that of the parameter matrix," is meant that there is no GUARANTEED statistical correlation between the parameter matrix and the error trend because the algorithm has no memory to make it possible for such to occur.

For such guarantee to exist, trend analysis would need to be theoretically integrated into back-propagation (which it is in RNN and Attention methods) and the theory would need to be applied to the algorithm design and implementation.

That is the second shortcoming and also the primary reason why biological neurons are remarkably more capable than ANNs.

* I meant to say above, "Kurt Gödel proved that there were truths that could not be proven in a formal system. (end parenthesis)"

Do you understand the lack of correlation now? How can an error trend be correlated to a matrix when the matrix adjustments are made from a history-blind error signal? Only by luck. LSTMs attempt to correct this beyond what earlier RNN schemes did, and converge faster, but the Attention devices (they claim and have not yet proven) do even better.

Your ranting. :)

I'm trying to share some information. Are you open to receiving it, or are you going to make it pointless to converse by nit-picking?

Okay. I hear that.

But I was first looking to see whether it was clear to you now.

Then my question would be, "Do you have any terms, analogies, or grammatical changes that would improve a sentence or paragraph, without compromising the importance of people learning the well-known 100 year old standard names for things?"

I'd be open to having footnotes explain terms, but I think people should do a web search for terms then don't know. There is nothing I use for terminology that is not in 100 online dictionaries. Before the web people had to turn pages in a dictionary. Now they can cut and paste the word into a search field. I won't pander to laziness (unless the membership here wants to pay me 200 usd/hr for being their technically-savvy personal secretary.

)

* they don't know

I did google "attenuation matrix neural network" and all the results that pop up just use Neural Networks in some way, and use the term "attenuation" in a completely different manner in another context

Now you're being absurd.

Its an adjective followed by a noun. We all know how that works.

You look up "attenuation".

no that's not how it works. I can learn very quickly what a "weight matrix" would mean in the context of neural networks by googling "weight matrix neural network"

but that doesn't work for attenuation matrix

If a reader does not know what a matrix is, then this is not a question for them, and perhaps they should get out their high school algebra book or take a continuing education class if they wish to engage in questions on the theory of neural networks.

Weight is not synonymous with attenuate.

Attenuate is the more specific and accurate term.

Can we take a break from this and discuss topology?

That was another term you did not like in the use I made of it.

it's a useless term if nobody uses it and nobody understands what you mean, even if it might technically be 0.01% more accurate if you dive deep into mathematical details of what's happening

when the entire AI community calls it weight matrix (or just parameters) all the time, there's no point in trying to push something else. ESPECIALLY not when there's nothing wrong with the existing terms. If "weight matrix" were confusing somehow, if it were ambiguous, if it could be confused with something else... yeah, sure, then we can try to push a new term

but there's nothing wrong with it, everyone will understand exactly what you mean if you just say "weight matrix"

As for topology, yeah, there we actually do have the problem I just mentioned does not exist for "weight matrix"; it's vague / ambiguous

If the intention of the writer is to awaken people to the fact that the numbers passing from activation layer to activation layer are in fact signals and that the matrix attenuates the signal, then the word attenuation has greater value than weight.

Nope, weight is perfectly clear; every number in the matrix is just a weight we assign to a connection, a measure of how "strong" that connection is

And you're not awakening anyone by using the term, you're just confusing people and making them click away from your post

It is my prerogative as an active member in this site and the author of this particular question, to attempt to illuminate the signal nature and information flow of the numbers. The theory makes more sense to the kinds of people I like to communicate to when the wider picture is brought into view.

I'm not into jargon because it creates silos of expertise and dissuades an interdisciplinary approach to research. If you don't like me for that reason, I'm truly saddened, because you are more able to comprehend the interdisciplinary approach than most, and that is needed in research.

I agree.

I should do it clearly and explicitly, just like all the other question writers. (sarcasm)

That we are not paid to write questions and can't monetize it in any way probably contributes to the low quality in clarity and explicitness. But that's true for all the questions on this site.

Also for blogging in general. (Make no mistake, SE is a collection of blog sites with a game objective. All the incentives are incentives of a game, and all the posts are those of a blog.)

most questions are very clear and explicit to me. They're short, to-the-point, ask precisely what they want to know (to the extent that an often-non-expert who is still learning is capable), and nothing else in between

The fact that you say that you want to "illuminate" people on the signal nature etc. indicates that you think most people are not yet sufficiently aware of that. By just replacing "weight matrix" with "attentuation matrix" in a rather large question, and not explicitly talking about it in any other way, you're not illuminating anyone.

I think you are not being truthful with yourself in that last statement. You may wish to review your comments on the posts of other Q&A authors than me before making such a blanket statement.

Consider this Q of yours: ai.stackexchange.com/questions/5398/…

I have no idea what you are talking about in it. I'd have to look up at least a dozen terms to grasp the question.

That "last statement" I referred to (while you were still typing :) ) is, "Most questions are very clearn and explicit to me."

That's fine. That question was specifically targeted at people intimately familiar with Deep Reinforcement Learning research of the past few years, it is specifically about that research, only anyone familiar with that research would be able to answer. For anyone familiar with that research, all the terms are commonly-used ones

This Q of min is specifically targeted to the inquisitive mind, like yours.

"attenuation matrix" is not a commonly-used term (again; I asked many times for just a single source, not seen anything yet), not for anyone

About 7,270 results (0.42 seconds) for "attenuation matrix" in Google's search.

What? We can't bring in general knowledge?

all in a wildly different context

If it isn't in a ML paper it isn't allowed?

Common, Dennis.

Is that your view of how science progresses?

it's allowed, but it's useless if your goal is to communicate effectively.

By disallowing new terminology and new ways of looking at things?

Is that what scientific and technological history bears?

Keep the terms to what we already know?

Don't think outside the box?

I KNOW that's not how you think.

if people introduce new terminology in science, they take the time to adequately define it, not just suddenly drop it in in place of existing terminology with zero explanation

Speak to the 7,270 uses in a search then.

because otherwise noone else will understand it

Maybe this is the reason many of the earlier contributers that had depth of knowledge like you and me left the site.

The readership has no interest in self-improvement, only quick fixes like on SO. And that's fine.

But if that's the truth this beta will never meet the criteria to breach the beta stage.

Don't you agree?

Seriously.

The nit-picking is the thing that's counterproductive, not the introduction of standard terminology from IMMEDIATELY ADJACENT FIELDS of information theory, mathematics, and control theory.

switching over to a different term that noone else in the field uses is not self-improvement. If I try to write "attenutation matrix" in my next paper when I mean "weight matrix", and I don't take the time to explain what I mean with it, I'll get some poor reviews about it and my paper won't get accepted

My writing does get accepted though. Not just here either. Professors pay me to ghost write.

I did, I admit, push the envelope here.

it's not nitpicking, nitpicking would be if I could easily understand what you meant but still decided to complain about a minor detail. Terminology that noone else in the field uses, when there is perfectly useful terminology available that everyone uses, is detrimental to the posts and in turn detrimental to the site

I'm looking through your Q&A and there are at least 50 cases where you wrote to a very narrow audience.

It is double standard, which is colloquially called nit-picking.

narrow audiences are fine, non-existing audiences are fairly useless

Maybe so, but, again, double standard.

Some of your questions have no answer that bears any comprehension of your question.

and I don't introduce my own new terminology or try to move terminology from other fields into this field. I just use the terminology that everyone in the field uses when it works fine

The one above has been up for 5 months, and you had to answer it finally.

It is not new terminology.

"or try to move terminology from other fields into this field"

It is allowed in any language to modify a noun with an adjective.

They don't have to have appeared before to make it a legitimate and comprehensible sentence.

yes, it's allowed, but useless if it causes the entire audience to stop understanding

Not useless.

You took the time to understand what the adjective meant in the context.

Now you know if you see that word, it MIGHT, but not necessarily refer to weights.

and I know I'll never use the term in the future because there are perfectly adequate alternatives available that other people will understand

You have full comprehension that the word relates to multiplication, and that a vector-matrix multiplication is what is at the front of each ANN layer.

I don't like the alternatives.

You do.

And that's fine.

If you wrote "Weights" or "Parameters", that's fine.

I don't care about what terms I like, I care about what terms other people will understand when talking to them

I think weights are very misleading.

not for anyone who has learned what a "Neural Network" is

or an "artificial neural network" to be pedantic

Actually, since you bring it up, "Neural Network," is also horrible terminology.

ANNs have very little in common with neural networks.

yes, I know. And again, I don't care when I'm talking to other people in AI, since they know what I mean. I'll try to care when I know I'm talking to a neuroscientist or biologist or whatever

The terminology in ML is inadequate and clumsy, and I'd knock it all down with a wrecking ball if I could.

But I can't.

I can only introduce terms from the science and engineering disciplines one at a time.

language is not static, language is dynamic, it changes all the time. And the meaning of words in natural language can depend on context, it's full of inconsistencies, that's why NLP is a difficult field of problems

I have deliberately sprinkled some of those terms around on this site, and I could use some help (or at least acceptance) to provide a clearer picture for people of what ANNs and their derivatives actually are.

so terms like "weight matrix" and "neural network" might not even be "wrong", it's just language changing

Most of the people on this site have no theoretical conception of what they are talking about and have admitted as much in several comments.

Weight matrix disguises the true nature that the ANN is a circuit.

Neural network is flat out lying by implying that the activation bear any real resemblance to neurons. That one is WORSE than jargon. It is marketing masquerading as science, and frankly, I hate that.

But I live with it and occasionally, in Q&A call attention to the differences without a less accusatory tone.

* with a less

Now with topology. I disagree with the use of topology as a term to describe the array sizes in the software.

Some people say, topology of [.... list of sizes ], and that is not aligned with ANY formal definition of what topology is.

If one could morph an activation or a parameter matrix or a connection between layers in a geometric representation of a network and it meant anything that would be a different scenario.

But changing the size of the block that represents an activation or a layer of them does not have any functional meaning.

For the term topology to have any meaning in ML or the larger AI picture, there must be things that ARE topological and things that ARE NOT topological.

This means that the minutia of array sizes is the prime candidate for NOT being a topological feature because, in the discrete space, an array size is the only possible analogy to stretching a shape.

I also do not agree that architecture = topology. They are not equal or equivalent. Architecture is the mapping of a solution to a problem and then to an execution environment.

Topology is precisely defined, again, in a dictionary. Although language changes, changing the meaning of topology to mean architecture or array size is dysfunctional.