« first day (3183 days earlier)      last day (1732 days later) » 
00:00 - 23:0023:00 - 00:00

enumaris
enumaris
11:00 PM
but why it's much faster on specialized hardware...I'm not certain...
 
JMac
JMac
Have the models been crippled by current technology?
Could just be intel trying to get ahead of the game I guess
 
enumaris
enumaris
@JMac a standard "artificial neuron" in a ANN operates quite differently than a neuron in a SNN
SNN's afaik, from very limited research, lag behind ANN's in that there's no equivalent of backpropagation for them
 
JMac
JMac
@enumaris Right, but could they not simulate this on regular computers as ACM mentioned?
 
enumaris
enumaris
since that process is not differentiable
yeah, you can
 
ACuriousMind
ACuriousMind
Well, if the computations required by such a spiking network are inconvenient on a CPU or GPU, then I understand why you want a specialized chip, but quick googling doesn't really enlighten my as to what exactly makes CPUs/GPUs so bad at these networks
 
enumaris
enumaris
11:01 PM
it's just a differential equation at the end of the day
kind of a specialized diff-eq...with the presence of a lot of dirac-deltas lol
(spikes = dirac deltas)
 
ACuriousMind
ACuriousMind
I mean, we have massively parallel GPUs because the CPUs can't handle the large data throughput and parallelism needed for graphics processing. What's it about these networks that neither the CPU nor the GPU are suited to them?
 
enumaris
enumaris
Yeah probably I won't understand the parts of this paper that specify why a standard CPU/GPU can't do this
I'll keep reading lol
 
JMac
JMac
@ACuriousMind Yeah that's what I'm trying to figure out too. Are they trying to provide tools to move the field forward or something?
 
enumaris
enumaris
They released the Loihi chip to academia or so they say
 
JMac
JMac
Because obviously this seems like a big research and even manufacturing investment for them, but if no one's asking for it that seems risky
 
enumaris
enumaris
11:04 PM
they just said the math is very different than "linear algebra" so...that's the level I got so far...
they already built the chips
at least prototypes
 
JMac
JMac
it sounds like they're moving to an actual manufactured product now though, right? maybe I misunderstood that part
oh research test chip
 
enumaris
enumaris
"When a neuron’s activation exceeds some threshold level, it
generates a spike message that is routed to a set of fanout
compartments contained in some number of destination cores. "
yeah, definitely research test chips atm
I could see how if all you wanted to do was simulate a SNN, then a hardware designed specifically for that could be more efficient than simulating one using software...
you'd have to simulate things like spike trains using arrays or something when you'd have a single neuron that can handle all of that in one of these chips...
 
ACuriousMind
ACuriousMind
Huh, this article from 2017 seems to say the main advantage of neuromorphic chips over simulating on a traditional chip wouldn't be speed but power-usage
 
enumaris
enumaris
possible
this paper did mention power usage a few times
but hey, with how much power bitcoin is using up right now...
 
JMac
JMac
That could be incredibly useful if we wanted to scale up practical machine learning
 
enumaris
enumaris
11:11 PM
in any case, it's very interesting! :D
 
JMac
JMac
Yeah, I'm starting to see why this might have been worth them adopting early.
 
enumaris
enumaris
if something akin to backprop happens for SNNs then I could see these chips as being very useful
apparently, in theory, a SNN can do anything a ANN can
 
b degnan
b degnan
@enumaris @Jmac I was sent here because I made neuron-ICs.
These are "in the wild" through DARPAs attentive binocular project.
The advantage is power, generally.
If anyone has question about the physics or circuits, I'll hang here for 30.
 
ACuriousMind
ACuriousMind
You were "sent here"? By divine inspiration?
 
b degnan
b degnan
No, one of my mates in the next lab has seen these ICs, and said that I should hop over, so I did. They're luring in here.
 
ACuriousMind
ACuriousMind
11:24 PM
Heh, interesting
So is there a way to understand why these designs use less power without a lot of detailed knowledge of chip design?
 
b degnan
b degnan
yes. they are charge integrators with a threshold. This means that if I have a small capacitor and a small current, I can have very long time constants with very low currents.
For instance, I can do key word recognition in 5nW (measured) with neuromophic approaches, or I can do it in 10mW in digital. It's also very application specific.
I am the "honorable mention" for doing most of the IC layout.
In analog, you get natural log mathematics for "free". Most of the natural processing algorithms require this mathematics. This is also very difficult for digital computers to do.
 
ACuriousMind
ACuriousMind
That makes some sense, thanks!
 
b degnan
b degnan
As "Moore's Law" (which isn't a law) has been in the cross hairs of the realities of physics, neuromorphic approaches have become more popular. Carver Mead (who coined the term Moore's Law, who also had a good chunk of Intel stock) wrote a book in 1989, "Analog VLSI and Neural Systems". That was the starter and is very approachable.
 
vzn
vzn
SNNs are "more biological" than ANNs in general but think its an open research question if they have any computational advantage. think markram behind the Human Brain Initiative in europe is a proponent. en.wikipedia.org/wiki/Human_Brain_Project humanbrainproject.eu/en/follow-hbp/events/…
 
ACuriousMind
ACuriousMind
@bdegnan So how programmable are these analog networks? Can you have a "generic" network or do you need to design the circuity with a specific application in mind?
 
b degnan
b degnan
11:36 PM
The analog approach works pretty well: youtube.com/watch?v=XVR5wEYkEGk If you can get to Tobi Delbruck's lab at ETH Zurich, you can see some silicon neuron-based cortex in action. I like the pencil balancing myself.
Very, that's the point. We use floating-gates to program them. We can also use the same techniques for unattended STDP learning. electronics.stackexchange.com/questions/240747/…
Halser's paper shows off some of these circuit techniques.
The floating gate (think FLASH but more bits) will start to generate hot-electron due to impact ionization when the drain moves out of a programmed area. This will change the weight of the device. This means that if the spike is infrequent, it'll have more weight (generally)
 
vzn
vzn
@bdegnan interesting, were they fabbed in US or overseas?
 
b degnan
b degnan
Depends. Delbruck uses XFAB in Germany (europe somewhere), Hasler uses TSMC out of Taiwan. I used IBM microsemi in Vermont/New York USA. I haven't made any ICs since IBM became Global Foundries in the neuromorphic space.
I made a single silicon neuron as a proof on concept on a 14nm SOI process from IBM. It worked, but you cannot publish over a single neuron. Patterned metals and other process quirks were challenging.
 
vzn
vzn
@bdegnan googling "darpa attentive binocular" does not seem to bring up anything but see nearby prjs eg "darpa + spiking neural networks" cacm.acm.org/news/…
 
b degnan
b degnan
Let me find the real name. One sec
 
vzn
vzn
@bdegnan yes maybe some of the idea/ enthusiasm is that possibly one SNN unit could require less electronics/ space/ power than one ANN (roughly speaking). and maybe there is more emphasis on analog (SNNs) vs digital (ANNs)...
 
b degnan
b degnan
11:44 PM
Cognitive Technology Threat Warning System
The Cognitive Technology Threat Warning System, otherwise known as (CT2WS), is a brain-computer interface designed to analyze sensory-data and then alert foot-soldiers to any possible threats, passive or direct. CT2WS is part of U.S. Department of Defense's effort to produce an efficient and working Network-centric infantryman. == Project == === Proposal === Proposed in early 2007, DARPA came to believe that a visual warning system could be produced and developed via an integration of technology and artificial intelligence. By combining discoveries in flat-field, wide-angle optics, larg...
How that actually was used, you put helmets on people and they watch movies at 24 frames/sec. You insert 12 images per second and then you read out the spikes when they see a "tank", "plane", etc.
All I can say is that the FPGA version of the neural networks always worked differently from the analog version with silicon neurons. They never came to the same conclusions. This does not mean they are incorrect, but the analog ones you can hook into a rat brain, the digital ones never really worked. That was outside of my direct work as it had to be done in France, but they used ICs out of the lab.
 
vzn
vzn
@bdegnan was that an academic research prj? or industry?
 
b degnan
b degnan
both. I was on the academic side. I made the ICs with a few other super great academics.
 
vzn
vzn
@bdegnan can you name the school?
 
b degnan
b degnan
Our partners were HRL and Northrup.
Georgia Tech in Atlanta, GA
 
vzn
vzn
ah yeah like to read Lance Fortnows + RJLiptons blogs... :)
 
b degnan
b degnan
11:50 PM
That's where I got my Ph.D. and did most of this work.
 
vzn
vzn
@bdegnan Phd in EE? is your Phd on the SNN?
 
b degnan
b degnan
If you look for me on google scholar, you'll see that I do unpopular, weird things that don't have much value, but if you look carefully, you'll see that I'm well funded for the ridiculous.
Yeah, I did temperature stability for these circuits.
no, You want Stephen Brinks Ph.D dissertation that I liked to above.
 
vzn
vzn
was your phd on the temperature stability? would be interested to look at it
 
b degnan
b degnan
I'm just a semiconductor physicist. I am a means to the ends for the neuromorphic people.
Yeah, it's terrible. I got in a fight with my advisor over a company that I started. There are two types of dissertations: excellent and completed. Mine is completed.
 
vzn
vzn
lol maybe there are 3 if you include ABD ... anyway its all impressive :)
 
b degnan
b degnan
11:54 PM
The experience was so terrible it's about to come out in a book. Everyone else thinks it's funny.
 
vzn
vzn
yikes sounds like phdcomics o_O ... (srsly, a book?)
 
b degnan
b degnan
much worse. My boss didn't sign my dissertation for two years after I defended because they needed someone to run the lab while they went to Thailand for some "updates". me: "what do I call you?" them: "Jennifer, and I want babies".
I got beaten up the Hell's Angel's. Got shit drunk with Lil' John and vomited in someone's Ferrari. Had the YAKUZA over from Japan for a party. I'm bad news.
2
The whole thing was terribly sad.
Parking deck fell and wrecked my car
 
vzn
vzn
this is all very cool and reminds me of this older chat session AMA idea we did a few yrs ago. think youd be perfect if you have some spare time sometime. for some reason they tended to work better in summers... physics.meta.stackexchange.com/questions/7783/…
 
b degnan
b degnan
I gotta run. I'm degs @ ece.gatech edu if anyone has any other questions.
 
vzn
vzn
@bdegnan ?!? lol was that all the same incident as the parking deck or...? have a good one/ thx for dropping by
 
00:00 - 23:0023:00 - 00:00

« first day (3183 days earlier)      last day (1732 days later) »