Automation And A.I. Systems will be the nexus for next generation Applications.

Join us on our journey of exploring the Techniques of these fields

Photo by Bram Janssens/Hemera / Getty Images

 

 

Artificial Neural Networks, Elixir, and You.

Artificial Neural Networks, Elixir, and You.

As I mentioned before on Automating The Future, I've been working for four years experimenting with the power of Artificial Neural Networks. I am convinced that the old way of developing software is coming to a close especially after witnessing first hand the capabilities of designing and training my own ANNs. In my previous post, I revealed a system known as VAGABOND to illustrate that its' completely possible to build a system without imperatively writing code instructions.

I was recently asked on Twitter how I produced this outcome, and what the code looked like, and the truth is there wasn't that much code outside of the ANN design. In fact setting up ANN's doesn't require that much code to solve a problem, all it needs is well-crafted data and a few coded mathematical formulas along with classifiers!! Remember, coding step by step instructions to produce an outcome will not scale in the future, so we have to look at system design from a whole new perspective. The next few posts will talk about how to design these types of systems from scratch because I believe it is going to be vital to understanding how these systems work if you are going to remain relevant in the tech field. I know first hand that AI is extremely powerful, and there is no need for the traditional programmer and all these complicated design patterns.

An Artificial Neural Network is driven by data, NOT IMPERATIVE CODE INSTRUCTIONS. This is something that you have to understand if you are to comprehend the world we are about to be living in. In fact, The companies that are still building systems that require imperative instructions programmed into them are going to get crushed by companies that are using this ANN based approach to solving their hard problems. Manual programming is just too slow and very error prone and is soon to be a relic of the past. Manual programming also requires too much babysitting and maintenance.

The imperative systems of today don't adapt, or learn, and are absolutely horrible at operating under uncertainty.  It's much easier to train a system via learning rules while allowing it to evolve itself, rather than manually commanding it to do what you want it to do while also being responsible for manually upgrading and maintaining it yourself. Coding really is no longer necessary,  yet there are tons of code schools teaching people to code when they really should be teaching people how Neural Networks are designed and trained. The day is coming when nobody will be hiring a programmer unless they have legacy systems they need you to look after. In the future, companies will likely hire experts in certain fields that will be responsible for training very powerful ANN's. It is very wise not to be in the dark as to how these systems work if you want to continue working in the tech field.

First, we will explore the ANN's and then we will finish up with why I've chosen Elixir as the go to technology for creating them. For me to truly understand something I like to first study the history of it. It turns out that the concept of ANN's was not at all developed by Computer Scientist, but was actually developed by both  Warren McCulloch a Neuroscientist & Walter Pitts a Mathematician in  the 1940's 

These two are given the credit for putting together the core model of how Neural Networks in the brain might work. Their research is fundamental in understanding the thought process needed to develop ANNs. It was a tremendous help for me reading their work when I wanted to wrap my head around this new approach to building information systems. You will find out that in the 40's this way of thinking fell by the wayside when traditional forms of computation took hold, and now it has come back in a big way. It is also important to note that the founders of this field were trained to think mathematically and scientifically, so building these types of systems will force you more into the field of Science & the Scientific Method, Statistics, Psychology, Biology,  and Mathematics. You'd likely start reading  more Life Science books than Technical books for ideas from nature because the ANN way of building systems is a complete 180 from what we've been doing for the past 60 years!

THE STRUCTURE OF A NEURON & ELIXIR

Artificial Neural Networks were essentially developed to mimic the human brain. This is a very large undertaking given the fact that we don't know much about how the human brain works. Even though our knowledge of the human brain is limited Science does know the basic structure of neurons and how they work together to pass signals back and forth in the brain. Neurons are at the heart of all brain activity. 

In order to develop an ANN, you first need to wrap your head around the 4 major parts of a Neuron. Surprisingly, these 4 major structures map perfectly to the structure of Elixir/Erlang's core technology!! I'm not sure if they did this purposely, but during my research, I was astounded by the similarities of the operations. 

How Elixir maps to Neurons.

How Elixir maps to Neurons.

It's' safe to say that if Elixir mirrors  similar characteristics of neurons then it's also safe to say that if developed correctly, Elixir Processes can be designed to mimic some of the known behaviors of neurons as well!!! This opens up a whole new world of automated systems. 

ESSENTIALS FOR BUILDING ANN's

Elixir/Erlang is positioned perfectly for building these types of cognitive systems. I'm not sure if the Erlang developers in the 80's did this intentionally, but as I explain the necessary components for building an ANN you will see why Elixir/Erlang eerily fits the bill. Trust me, I tried building in Python and have recently made the switch over to Elixir/Erlang because it is designed from the ground up for this type of problem-solving. Here is the list of things you need to have to build an ANN from scratch

  1. You absolutely need Parallelism & Concurrency.
  2. You need the ability to Distribute across multiple Nodes without that much hassle.
  3. FAULT TOLERANCE!!!! This is an absolute must, especially with hundreds of thousands of processes communicating concurrently.
  4. Real Time updating capabilities via channels.
  5. The ability to handle a large influx of data without being overloaded at all (Thank You GenStage!!!)
  6. Pattern Matching capabilities
  7. Ability to write mathematical functions like SUM, MAX, AND, OR, as well as sigmoid transfer and activation functions. (More on their implementation in coming posts)
  8. A very powerful machine with multiple cores for maximum fun. I've recently been training ANN's with one of these.

These are just a few points I've found to be handy while playing  around with ANN's. Your needs and desires might differ the deeper you get into playing with them.

THE STRUCTURE OF AN ANN & ELIXIR PROCESSES

Let's dive into how a simple ANN is structured. Remember from your ancient Computer Science class that every input has to have an output. This notion still holds true for ANN's. In fact, there is a 1 to 1 relationship between Neurons in regards to their input and output. If I put in an input to one neuron I should only get one output value from that same neuron. This communication between two or more processes can be thought of as a neuron connection. We can simulate a connection in Elixir by calling "Process.link/1" which is what is actually being referenced when you make the following OTP calls in Elixir.....

#Connection to another Process/Neuron via GenServer
GenServer.start_link(GenServerName, {}, options \\ [])

#Connection to a Process/Neuron AKA Neuron via Agents
Agent.start_link(AgentName, function, {}, options \\ [])

#Connection to a Process/Neuron via Supervisors
Supervisor.start_link(SupervisorName, {}, options \\ [])

Artificial Neural Networks always have 3 overall layers to them. The input layer, the hidden layer, and the output layer. The hidden layer can have as many connections as it needs to solve a specific problem. I've found that the harder the problem is I wanted my system to be able to solve the more connections it created in the hidden layer.

A very simple ANN looks a little something like this...

A simple Artificial Neural Network.

A simple Artificial Neural Network.

 

You can view the lines connecting the processes/neurons in the hidden layer as messages being passed back and forth along the synapses. This hidden layer uses 7 components to generate a self-learning type of system.

  1. Weighting Factors 
  2. A Summation Function
  3. A Transfer/Activation Function
  4. Scaling and Limiting
  5. Output Function
  6. Error Function (Back-Propagation)
  7. Learning Function

I've found that these 7 components are at the core of all learning based systems. This post will cover the first 2 in detail.

CALCULATING INPUTS FOR A NEURON

We brought up inputs, now it is time to explain exactly what they are. In my research, I've found that the Feed Forward Neural Network is the most commonly used Neural Network. That is, there is always a biased input of 1.0 passed throughout the network to help make the weighting process calculation easier. 

Let's say we have a Neuron/Process that is called N, and it is receiving 3 input messages. i1, i2, i3. Our goal is to calculate what the output of N will be using a standard formula known as the Calculation For Activation.  This essentially wakes up the Neuron, but before we can do that, we need to compute the inputs and weights summation which is the argument for the activation function.

i1, i2, i3 are called inputs. All inputs have associated weights. Weights are defined as a measure of a inputs connection strength. These strengths are modified via learning rules, which we've talked about previously on Automating The Future. Weights are statistical numbers between -1 and 1. Weight values are extremely important in generating the output for a specific outcome. In a Feed-Forward network, one input will always be 1.0 or 1, and the other two are usually random. Looking at our example, each one of our "i" inputs will have an associated weight w1, w2, w3. In Elixir we can express this message in code like ...

Weights & inputs for our Neuron example

Weights & inputs for our Neuron example

Next step for us is to apply the summation of our inputs and weights. For each one of the inputs, we want them multiplied by its' associated weight. Then for all those results we need to sum them up to get one output.  This calculation is pretty simple to express in a functional language like Elixir utilizing "Enum.map_reduce/3". However, in order to make everything crystal clear I wrote out a summation function the long way so that it is fully understood what is happening.

The long way to calculate the summation for our inputs and weights.

The long way to calculate the summation for our inputs and weights.

So lets see what our summation is for the above inputs and weights...

The summation of our inputs and weights.

The summation of our inputs and weights.

Great! Understanding this basic step is vital to comprehending how ANN's work as it is the first step. The next step is passing this value to our activation function. Before we can do that we have to understand what an activation function is and what it is for. That explanation is reserved for the next post. 

The Power of Activating The Artificial Neuron

The Power of Activating The Artificial Neuron

The People Suffer From Lack of Automation.

The People Suffer From Lack of Automation.