Neural Networking – Fact or Fiction..?

Hiya Folks,

First of all why is Neural Networking perhaps the only way forward for the next generation of AI applications?

Digital processing is inherently suited to deterministic problem solving, i.e. the problem can be well bounded with a firm set of requirements. Indeed look at any software development methodology in any software house and you’ll find very strict processes. These take the software from requirements definition, high/low level design coding and feature and verification test. The names for the various phases may differ but the overall principle is identical.

If a problem is not deterministic then we have a problem with this approach. The real issue for developing more robust AI software and hardware based systems is that a lot of real world situations can not readily modeled in the digital. This problem though has been solved elsewhere though. After billions of years of evolution nature found a way. The answer is the animal brain. Failure of the “requirements capture” or an un-encountered “error leg” is not a delay in the final product but more than likely death. This has been the incentive to get it right.

Nature seldom has a single all responsible element and very seldom processes in serial. In nature complexity is always built up through the collective effects of many simple units with redundancy built in. A computer is usually the exact opposite of this.

The first real attempts to crack the non deterministic nature of problem solving are Neural Networks. These are a new paradigm for computing based on the parallel architecture of animal brains. They are a form of multiprocessor computer system; with simple processing elements, a high degree of interconnection and adaptive interaction between elements. Early research in Neural Networks produced some very promising results. Indeed my colleagues and I have used modifications of basic Neural Networks to solve problems as diverse as multi-parameter minimization, 3D image reconstruction from stereo picture, event classification and even prediction of sunspot cycles.

My question is that Neural Networking seems to be losing its ability to reach critical mass. In the mid 1990’s there was a glut of companies founded on NN theories, striving to reach the ultimate goal of truly intelligent systems. Some of these software companies have found limited success:

Neurodynamics
Convera
AND Corporation
ERA technology
Gensym
Imagination Engines
Invention Machine
Knowmatch
Ward Systems

Their questionable financial successes aside, where are the noticeable benefits to mankind? Considering this technology has been discussed and developed for almost 20 years how come NN have not truly been able to gain a foothold in mainstream applications and automated processes? Where are the killer applications that will present untold riches to companies able to manipulate and interpret data in similar ways to our own organic thinking processes? Maybe it’s not possible to mimic organic parallel processes in an inherently serial environment?

 

Log in or Sign up to hide all adverts.

Hi,
Welcome to sciforums.
ANNs were introduced a long long time back as far as i can remember.But it was never truely implemented.Since R&D hasnt been always on the right path.Take for instance Designing.Many think Neurons as a Transform function,some percieve a neuron as an object.This difference of opinion has somewhat given a setback to the development of ANNs.But i am optimistic that it will pick up.

I think Expert Systems are already working on the concepts of Neural Network Simulations,Vraious companies are already using it.While at Installation of VSATS,We saw one such System which was used for Identification in Telestra(Australia).It worked on OOD concept and used weight theory for its working.It had been installed just a couple of months ago and had shown remarkable promise to the Company in DBMS as well as ID servicing.



thanks.
bye!

 

Log in or Sign up to hide all adverts.

There are two problems with neural networks.
1. Inputs have weights.
2. Answers are for humans.

The neural name is wrong also because they do not match neurons.
A neuron has one output and many inputs. It matches the idea of a mathematical function.

In 1959 Marvin Minsky infected me with AI and I kept waiting for those bright guys to do something.
Years later, after all my mentors died, I realised the AI people were trying to build an abstract idea. Even if the problem was solved it would be stuck in a computer.
In 1994 I started at the bottom, looking for the least brain. I needed something basic that would scale. I discovered nodes and cables, which is a network without weights. The data in the network are 8 bit variables, or a base 256 digit. This is because each node is a microcontroller. Each node has one or two inputs and one output. With two inputs you can build n input trees.
There are at present about 20 node types.

This system makes an algebra and a language you can use to build any structure you can write. The language maps to nodes so you program is description of a machine.
See some of my other posts here. I have a really hard time explaining this to people.

Harold

 

Log in or Sign up to hide all adverts.

I was thinking about writing this previously, and decided that it would be a good idea just to put it forwards.

Have you thought about how a single Output can be "Percieved" as multiple outputs. The idea being that a "field" is generated during an output, and the intensity of the field can be different.

Due to the intensity of the field, surrounding neuron clusters can have a "doppler Effect", in essence a kind of Non-locality of data.

Could that not be apart of a true neurological structure?

 

Yes.But i dont get it.What has other Neurons Got to do with what Inputs the Single neuron is recieving?...

I thought all Neurons are considered to Be Independent...



bye!

 

A neuron is a tree of computational elements. Neurons are made of trees, trees are made of neurons. Neurons can be added to the brain as if they were numbers. In fact brain is simply a list of neurons, including sensors and motors.
Think of the neuron as an input output device. It has n inputs, and one output. (The brain has Q inputs and Y outputs. Divide Q by Y and you have the number of neurons.)
Each neuron is a function that reads n inputs, ripples them through n/2 processors and outputs the result. Each neuron has a purpose in the brain. The brain structure matters. It is not randomly connected.
The number of inputs to a neuron (n) can have n! different values. During its life and neuron may only see n values. That leaves two ways to learn something new.
1. rewire the system for new functions.
2. Get a new input and incorporate it in the system
Both of these happen in a big, wet brain.
Both of these can be done with hardware.

 

Hmmm...What if...

So let’s assume that we have developed our ANN architecture, how can I maximize the learning abilities? After all I want to spend the minimum amount of time training my networks. In fact they should be teaching themselves. An obvious answer might be to cross ANNs with Competitive Learning processes; this should encourage incremental development so that new strategies are developed by the ANN.

Lets take Tic-tac-toe, I create 10 ANNs, and play them against themselves typically its opponent adjusts its abilities and discovers new strategies of its own. I like the principle of a strategic “arms race” but I was always played with toy soldiers as a child. So as we have shown the number of good moves increases over each generation and the overall ability of the learners eventually reaches an optimal level, in our case the ability to win every game or at least draw.

Interest for using competition in machine learning tasks stems from a desire for a program to discover the strategic nuances of a complex task directly from the first principles of interaction. Appropriate complex structures arising purely from the “physics” of the task environment would be the ultimate validation of machine learning capability. By making this an evolutionary process.

FYI Samuel (first in1956, then in 1966) was the first to use a self-practicing model in his checkers learning experiments. His program induced a linear function that predicted the utility of particular checker board configurations. However it seems that he had some flaws in the first experiment. So in 1966 he created a more complicated competitive method with two competitors that updated their respective evaluation functions at differing rates. Even though this more complicated method uses two separate learners, it is still considered competitive since the two learners comprise a single composite learner that is teaching itself.

Anyway back to my point, in 1992 Tesauro describes a connectionist network, called TD-Gammon, which acquires a valuation function that played a very good game of backgammon. The difference here is that we have mulitple ANNS all playing each other.

Anyway TD-Gammon uses it’s developing network as both opponents to play a game. Once the game is completed the weights of the network are then updated using a temporal difference method (Sutton 1988). The game provides both positive and negative feedback to the network since it plays both sides. Tesauro reports that TD-Gammon plays better than his previous best network backgammon player, Neurogammon (Tesauro 1990). Either way it shows that Backgammon was another game of choice for investigating competitive learning.

The crux of competitive learning is the assumption that a learner improves itself through practice with itself or another entity, identifying flaws in its own training that even it can exploit in the task. Once repairs are made, the learner moves closer to an adequate concept without a prior knowledge signaling in which direction to move. Repeating competitive learning, it is thought, incrementally improves performance and progresses the learner through improving strategic phases. In essence, competition is expected to force the performance of the learner toward more and more sound strategies. As each new strategic induction is made, it is employed both to its own benefit and demise, fueling further refinements to the concept.

While the principle of competitive learning appears sound, the assumption that the learner steadily improves toward the goal is perhaps overly simplistic. Epstein (1992) argues that a self competing learner alone does not provide sufficient feedback to acquire adequate concepts in some learning environments. Of course Epstein never employed an evolutionary function that rewarded correct and proficient behavior.

Her experiments on two player deterministic games, such as Tic Tac Toe, showed that competition acquires concepts consistent in ability as those induced using a teacher playing optimally about half of the time and randomly otherwise. In our case we can show a linear improvement and we always reach the most optimal performance for our NNs playing tic-tac-toe.

Where Epstein failed, we will succeed. She concluded that competition is a bit myopic about the identification of incorrect inductions in the learner. This creates the potential to under explore the available moves unless there were specific attributes in the task to encourage otherwise.

In any learning system, the implicit “goal” is to locate a strategy such that no feedback received forces the training algorithm to change it. In other words, the concept is stable with respect to the teacher and method of reward.

However what we need our ANNs to experience are the consequences of playing a nondeterministic game. Because backgammon requires a dice roll on every move, the ANN entity is forced into a wider variety of strategic situations than if the task was deterministic. The dice in backgammon creates unpredictable advantages for each ANN entity that we cannot predict, reducing the competing DANIs ability to respond. Also the roll of the dice occasionally forces play into board configurations that have never been found in any previous games, and consequently provide the feedback to refine the network.

FYI, Goldberg (1989a) developed Tournament selection, a competitive selection method, it seems to be popular in genetic algorithms and evolutionary programming (Fogel 1992a) I missed it previously as I was searching thru just the NN papers. The theory states that once the fitness of each individual is discovered, the parents of the next selection are selected through comparison. In a k-competition in genetic algorithms (Goldberg 1989a), the fitness of k randomly chosen members of the population are compared. The competitor with the highest fitness value is declared the winner and copied into the next population. This process is repeated with replacement until the next population is filled. In the tournament selection of evolutionary programming, a competition score for each population member is created by comparing its fitness with k-1 other population members and awarding a point for each competitor with a lower fitness.

Also Axelrod (1987) experiments with both an independent and a competitive fitness function to evolve strategies for the Iterated Prisoner’s Dilemma (Luce and Raiffa 1957; Rapaport 1966; Axelrod 1984). For the independent fitness function, Axelrod uses a weighted sum of the scores against eight pre-selected strategies, where the weights and representative strategies were selected on the basis of knowledge gained from previous experiments (Axelrod 1984). In the competitive fitness function, Axelrod (1987) tests every population member against every other population member, which presents an adaptive developmental environment for the population. Axelrod’s experiments attempted to determine if a cooperative strategy for the Iterated Prisoner’s Dilemma could evolve from random initial conditions in the population. Because each member of the population always competes against every other member of the population, there is ample opportunity for reciprocation between differing strategies.

Has anyone here explored these avenues for developing ANNs? Any success?

 

Before we can explore any of these things we need to at least know how to build the brain of a microbe. That is not as easy as you think, but if you cannot describe that, how can you describe bigger brains and learning and other buzz words?
Your time will not be wasted. Brains have no scale so what you learn will be universal.
Think about this. If you are interested I will build one for you here.

 

Brain for hire?

Hiya HL,

So what do I get for my money? Your website seems quite vague, I would like a little more information before committing myself.

How is your neural network architecture layered?
How many interconnected nodes contain an activation function?
What inputs are available?
How many hidden layers can you interconnect, are they all weighted connections?
How do you modify these weights?
How do you make sure that you have representative data?

I have found that the error minimization space is seldom a smooth and so the learning process, which is usually some form of gradient search, can be tortuous due to local minima in current ANN architectures.

 

This subsumes any other networks. (I can emulate them)
The idea here is to go from goals, motor actions, back to sensors, incrementally.
Seekingfood = detectingfood - nofood.
Then build from and define detectingfood, then nofood.
When you are done, you have the code for a brain function.
When you are done every variable in the system has been defined and you have a parts list for your machine's brain.

Hal Trees are made from Hal nodes. Hal nodes are computers doing a single function forever.
Any two input function will form an n input tree. Start at the motor or axon end with a node. Add two nodes to that. Add four nodes to those and on up to n.

There are two types of Hal trees. Full trees are fully connected and sparse trees are singly connected. (Need a picture here.)

Full trees, in addition to the nodal functions, multiply the inputs by the binomial coefficients.

A = S1
B = S1 + S2
C = S1 + 2*S2 + S3
D = S1 + 3*S2 + 3*S3 + S4
...

Sparse trees are:
A = S1
B = S1 + S2
C = S1 + S2 + S2
...
+ here means any of the 9 tree functions.

The tree functions are:
+ PNode ; adds I + R
- CNode ; subracts I - R
o ONode ; gets Max(I,R)
a ANode ; gets Min(I,R)
is ISNode ; detects equality I = R
waseq WaseqNode ; detects if ever I was equal R
wasgt WasgtNode ; detects if ever I was greater than R
waslt WasltNode ; detects if ever I was less then R

There are switch nodes that switch R on condition of I

There are function nodes that work on single data streams.
d DNode ; O = d(I)
i INode ; O = i()
n NNode ; O = n(i)
not notNode ; O = not(I)

There is a lot here. The idea is to have a function availible to make any structure you can imagine and write down.
There is an algebra that you can use as if you were writing a computer program.

This is a system that you must learn. It is for making things.

My web site has many places you can talk to me. I ask for comments. I have had a verbose site up for some time and this is a new one that simply sells design and brain kits.

I am writing a new book on this and this is testing the audience.

Thanks,

Harold

 

cool thread. bump.

Please Register or Log in to view the hidden image!

 

I've heard that genetic programming is used more and more instead of neural nets...is it true ?

 

Different animals.

Genetic programming sets up a lot of programs, gives them a score and lets them fight it out. No new programs are made here.

Neural nets, as currently used, are simply decoders. For a given input, weights are adjusted to give a known output.

Neither of these work, which is why I created Hal algebra which allows you to construct trees that match neurons, inputs to output.

This has not made it to academia yet.

 

Actually although genetics and Neural Networks for system development are two different kettles of fish, some branches I believe are appearing that link the two.

On is the fact that a Genetic system could be created instead of the usual siliconchip and semiconductors.

One thing that is being researched is "Quibit" computer systems that store data and process data to quantum superpositioning of atoms.

The understandible usage here is that a genetic system could be used to house the atoms, and allow a communication structure to grow.

This type of usage however sounds quite scifi, and you probably won't hear too much of yet.

There are however more studies in the field of cybernetics to work out how to link genetics to machine, which could prove useful to.

As for Zion since my response has been so long.

My explanation about how the brain works is that information doesn't just transverse the stems. Imagine a tree it has branchs, and the current assumption is that when you send a signal it has to find it's way to the trunk to then find it's way to the right branch for delivery.

I believe the reality is that the branches give out a wave formation that can be felt by nearby branches, so the information can be worked on by clusters that aren't actually connected.

(This theory can be proven if you look at a certain condition that can occur where the Brain has a particular point that is suppose to connect hemispheres, in fact has a disattachment, but the person can still function as if the communication paths were there.)

 

Yes,
Neurons are trees. Sets of neurons are trees. The data comes in simultainously on the leaves and is functionally combined to the trunk (axon)
It is time that ties all this together. If there are n inputs, there are n time ticks to the axon. Look at all the inputs at one time and you have another curve that sweeps in n steps through the neuron and can be any of several functions.
Simple differences, provide n derivatives of the input curve. This curve is repeated every time tick and all the data is from other neurons with their own time.

This must be, else there is no way to compute anything.

Each neuron is stupid, but it can have memory of conditions of inputs.

There are two main types of neural trees. Command trees collect data from sensors to brain to motors. Collection trees collect data back from the motors and other sensors. There is an architecture here, which is why we do not have sentient networks or railroads.

I did all this in 1994 and wrote a book on it.
Book is "Brains for Machines/ Machines for Brains"