Building synthetic brains capable of human level discovery and invention...

Currently, the IEI patent suite covers four areas of artificial neural networks that are essential for the building of synthetic brains. They are (1) Device for the Autonomous Generation of Useful Information, (2) Non-Algorithmic Neural Networks, (3) Data Scanning, and (4) Device Prototyping. Collectively, these patents put IEI in a unique position, to build synthetic brains capable of human level discovery and invention.

1.0 Device for the Autonomous Generation of Useful Information (Creativity Machines).

The first of these patent groups deals with how to stimulate trained neural networks to make them generate ideas and plans of action that are outside of their direct experience. As you may recall, traditional artificial neural networks absorb memories and relationships. However, until stimulated with various kinds of unintelligent forms of noise, they perform deterministically, transforming input patterns to output patterns. When stimulated at just the right noise levels, they begin to generate new potential ideas or plans of action that are generalized from their training patterns.

Whereas the discovery of just how to adjust the noise level within a trained neural network to produce new ideas is a significant scientific finding, a viable patent was not achieved until a critic algorithm was added, whether heuristic or neural network based, to monitor for the very best of the ideas emerging from the perturbed network. This is the preferred embodiment of this invention, a 'dreaming' network or 'imagination engine' that is monitored by another constantly vigilant algorithm that we appropriately call an 'alert associative center'. So as to accelerate the convergence toward the optimal concepts, this critic is allowed control over the perturbations applied to the imagination engine.

From a legal perspective, the inventor had reduced to practice an exhaustive list of schemes for perturbing a trained artificial neural network to generate useful information, whether stochastically or systematically driving the inputs of a network or hidden layers of a neural architecture. Furthermore, the stage is set for brain-like architectures capable of creativity, by teaching parallel, distributed neural cascades consisting of multiple imagination engines and alert associative centers. Such compound Creativity Machines are now capable of carrying out juxtapositional invention wherein one imagination engine thinks 'wheel', another 'axel', and a critic network makes the pivotal association with some form of wheeled transportation.

Finally, this patent forms the basis for creating conscious machines, in that it emulates the chief cognitive circuit within the brain, the thalamo-cortical or thalamic-cortical loop, wherein the cortex generates a relentless stream of memories and ideas, as the thalamus, often called the "eyeball" within the brain, is on the lookout for notions that are of interest or value to it. Essentially, one cannot build a synthetic brain without this essential neural mechanism.

2.0 Non-Algorithmic Neural Networks (STANNOs)

When IEI became involved in control systems and robotics in 1996, it quickly became evident that Creativity Machines needed to learn from their own mistakes and successes. What was needed were imagination engines and critics that could cooperatively invent action plans, implement them, and then judge whether they had achieved the intended results. If they had succeeded, reinforcement learning would need to take place in all networks. If they failed in meeting their objectives, at least the critic networks needed to be trained to recognize the negative results.

Realizing that artificial neural networks are customarily trained one at a time, using what is called a training algorithm, typically several pages of C-code, it became exceeding difficult to train two or more neural networks simultaneously within Creativity Machine architectures. We required a neural network that came bundled within its own training algorithm that would be capable of training in situ within Creativity Machines. Ironically, it was a Creativity Machine that autonomously designed a brand new form of neural network architecture that consisted of a trainee network intimately intertwined with a trainer network. Since both algorithms were implicit, taking the form of numerical connection weights, and not explicit rules, these networks were termed 'non-algorithmic'. Encapsulating these purely connectionist structures within a class wrapper, we could instantiate ultra-fast and efficient neural network objects to sizes and numbers that were limited only by memory.

It is important to note that although these STANNO patents are couched in terms of spreadsheet-based neural networks, that most of the paradigms that form the subject matter of these patents apply to other computer implementations of them. In fact many of the independent claims discuss these principles regardless of whether they are used in a spreadsheet environment or not. One way to think of this process is that Microsoft Excel became a convenient environment in which to pioneer these concepts.

Advanced STANNOs incorporate many of these patented principles, but also exercise a number of trade secrets to make them even more flexible, fast, and efficient.

3.0 Database Scanning System

A number of new pattern recognition techniques were prototyped using techniques that were, in turn, prototyped using spreadsheet-based development environments. In these patents the term 'database' is used in the general sense of any repository on a computer for storing temporarily or permanently, any kind of data pattern. This terminology would include traditional databases such as Excel or SQL, or storage buffers associated with cameras, or other high speed data acquisition devices.

The most important of the claims within these patents have to do with the use of so-called auto-associative networks that train upon patterns that are somehow interrelated (i.e., multiple camera views of an object, or states of some system hardware that is to be controlled). Once trained upon numerous examples of such genre, they can quickly determine outliers that are non-representative of the group of exemplars previously shown to the network in training. Alternately, by process of elimination, it can also identify patterns that are representative of a genre. 

When such group membership filters are implemented via STANNOs, they can be instantiated on platforms as humble as PCs, and still contain millions of inputs, millions of outputs, and a significant number of hidden layer nodes. As a result, we can now connect to cameras and perform anomaly detection, target classification, and training using on the order of a million bytes of information and millisecond time scales.

Furthermore, this patent teaches the essential components of what IEI calls 'foveational systems', simple, yet elegant two network systems that allow machines to scan data in a manner similar to the way the eye scans its environenment. Essentially an imagination engine generates a series of coordinates to focus its attention on. Other neural networks may then be added to look at whatever the former is focusing on. The feedback mechanism of the Creativity Machine may then be employed to modulate the magnitude of perturbation, that is, chaotically driving the position of the attention window depending upon a critic's 'interest' in that window's content. In this way, just like the saccade movement of the eye, motion is chaotic until it clips a piece of what is being sought, or something of general interest to the system.

4.0 Device Prototyping

Generally, neural network practitioners acknowledge that when building neural networks, one must use so-called activation, or transfer functions that are mathematically well-behaved and can be expressed in some closed, analytic form (i.e., sigmoids and hyperbolic tangents). STANNOs are quite different and break some of these long standing rules, in that they can employ arbitrarily complex activation functions. In fact, STANNOs may in fact use other STANNOs as their individual processing units, resulting in STANNOs of STANNOs, or what we call SuperNets. These are not ordinary neural network cascades, but organic cascades in which all component networks train in parallel, and, while so doing, autonomously connect themselves into vastly complex neural architectures.

One extremely valuable task that such SuperNets may perform is what we call 'device prototyping'. To illustrate, consider a collection of neural networks within the hidden layer of such a compound net, each of which has been pre-trained to simulate the behavior of various electronic components. Some may be diode simulations, other capacitor or logic gate models. When the overall system, represented by the SuperNet, is trained upon patterns representing the overall input-output characteristics for the intended electronic device, the STANNO-based simulations will connect themselves into the necessary topology to achieve such function, in the process possibly eroding away connections to device simulations that are unnecessary for device function.

In the same manner, we may allow vast swarms of such STANNOs, and STANNO-based Creativity Machines, to automatically connect themselves into brain-like structures. In this way, very robust machine vision systems may autonomously connect themselves into the equivalent of vision pathways of the brain. Similarly, robotic brains made of STANNOs could spontaneously organize themselves to be capable of devising plans of action that were totally unanticipated by the robot's human creators.

In the language made popular recently by those overlooking these patents, the SuperNet is a hierarchical cascade. However, the cascades these authors speak of are constructed by hand. In stark contrast, SuperNets build themselves.