7. The Network Approach
Influenced by the principles of operation and organization of real-world brains.
Connectionism tries to understand how the mind performs certain kinds of operations via
the construction of an artificial neural network (ANN) - a computer simulation of how
populations of actual neurons perform tasks.
Artificial Neural Networks
Traditional computers are serial processors: perform one computation at a time. The brain,
as well as ANNs, are parallel distributed processors: large numbers of computing units
perform their calculations in parallel. Knowledge-based approach: One conceptualizes the
problem and its solution in terms of symbols and transformations on the symbols (used a lot
in AI). Behavior-based approach: A network is allowed to produce a solution on its own.
This does not involve the use of symbols (ANNs). Representations are inherent in ANNs but
do not exist in them in the form of symbols. They exist in most networks as a pattern of
activation among the network’s elements - distributed representation. Local
representation: in the form of activation in a single node in a network.
Pro: They are capable of learning —> adaptively change their responses over time as they
are presented with new information (but also possible in machines that use symbolic
methods).
Characteristics of ANNs:
● Only exist as software simulations that are run on a computer
● Each neuron is represented as a node, and the connections between nodes are
represented as links.
● Signal node: activation value —> runs along the link that connects it to another
node(s)
● Input > threshold value —> fire
● Links have weights: specify the strength of a link. Higher value, higher weight.
Early conceptions of Neural Networks
First researchers to propose how biological networks might function: McCulloch and Pitts,
1943. They assumed each neuron had a binary output, it could either send out a signal or
not send out a signal. Donald O. Hebb (1949) was the first to propose how changes among
neurons might explain learning —> Hebb rule: when one cell repeatedly activates another,
the strength of the connection between two cells is increased. He defined 2 types of cell
groupings:
1. Cell assembly: a small group of neurons that repeatedly stimulate one another
2. Phase sequence: a group of connected cell assemblies that fire synchronously or
nearly synchronously
Rosenblatt introduced in 1958 the perceptron: neural nets designed to detect and recognize
patterned information about the world, store this information, and use it in some fashion.
They also learn from experience: can modify their connection strengths by comparing their
actual output with a desired output called the teacher.
Back Propagation and Convergent Dynamics
,Three layer network:
1. Input layer - a representation of the stimulus is presented
2. Hidden layer - feeds activation energy to an output layer
3. Output layer - generates a representation of the response
Error signal: the difference between the actual and the desired outputs. The network uses
the error signal to modify the weights of the links. The kind of training based on error
feedback is called the generalized delta rule or the back-propagation learning model.
NETtalk Is an ANN designed to read written English. Presented written letters —>
pronounces them —> fed to a speech synthesizer for the production of the sounds. System
consists of 3 layers.
Connectionist Approach:
Pro:
● The similarity between network models and real-life neural networks: b iological
plausibility.
○ Artificial Networks share general structural and functional correlates with
biological networks
○ Artificial networks are capable of learning
○ Artificial networks react to damage in the same way that human brains do:
neural networks demonstrate graceful degradation - gradual decrease in
performance with increased damage to the network. Small amounts of
damage —> small reductions in performance.
● Displays interference (2 sets of information are similar in content and interfere with
each other) and generalization (represented by the ability to apply a learned rule to
a novel situation)
Con:
● Biological plausibility should also be viewed as problematic
○ Real neurons are massively parallel, it is not yet possible to simulate parallel
processing of this magnitude.
○ Most networks show a convergent dynamics approach, the activity of such a
network eventually dies down and reaches a stable state. This is not the case
for brain activity. Real neural networks are oscillatory and chaotic.
● Networks may have inadequate learning rules
○ Stability-plasticity dilemma: states that a network should be plastic enough
to store novel input patterns; at the same time, it should be stable enough to
prevent previously encoded patterns form being erased. The fact that ANNs
show evidence of being caught in this dilemma is useful because it may offer
some insights into human interference.
○ Catastrophic interference: occurs in instances in which a network has
learned to recognize a set of patterns and then is called on to learn a new set.
The newly learned patterns suddenly and completely erase the network’s
memory of the original patterns.
○ In supervised networks, a “teacher” is necessary for the network to learn.
But where does this teacher come from?
,Semantic Networks
In semantic networks each node has a specific meaning and, therefore, employs local
representation of concepts. Has been adopted by cognitive psychologists as a way to
explain the organization and retrieval of information in long-term memory.
Characteristics of Semantic Networks:
● A node’s activity can spread outward along links to activate other nodes, these nodes
can then activate still others: spreading activation. Is thought to underlie retrieval of
information from long-term memory. Alternate associations that facilitate recall are
also called retrieval cues.
● The distance between two nodes is determined by their degree of relatedness.
● Priming: the processing of a stimulus is facilitated by the network’s prior exposure to
a related stimulus.
Hierarchical Semantic Network
Study by Collins and Quillian suggests that semantic networks may have a h ierarchical
organization, with different levels representing concepts ranging from the most abstract
down to the most concrete. They used a sentence verification task.
1. Superordinate category: animals —> eat food, breathe
2. Ordinate categories: birds, cats —> can fly, purr
3. Subordinate categories: Canary, Alleycat —> can sing, is yellow
A canary is an animal —> longer reaction time than ‘A canary is a bird/a canary’
“isa” and “has a” link, bird “isa” animal, bird “hasa" feathers
Con:
● Concepts may be represented by prototypes that represent generic or idealized
versions of those concepts.
● Principle of cognitive economy: nodes should not have to be coded for more times
than is necessary. Seems to work better in principle than in reality.
Propositional Semantic Networks
ACT* is a hybrid model: it specifies how multiple memory systems interact and how explicit
knowledge is represented. A proposition is the smallest unit of knowledge that can be
verified. Propositional networks allow for a greater variety of relationships among concepts.
An agent link specifies the subject of the proposition, an object link denotes the object or
thing to which the action is directed. The relation link characterizes the relation between the
agent and the object. Anderson’s ACT* model can also account for the specific memories
each of us has as part of our experience. His model does this via its creation of 2 classes of
nodes: type node; corresponds to an entire category (‘dogs’), t oken nodes; correspond to
specific instances or specific items within a category (“Fido”).
Semantic Networks Evaluation:
Con:
● T.O.T. phenomenon: ‘tip of the tongue’. Semantic Networks cannot easily explain
these sort of retrieval blocks.
● The opposite; the situation in which we can successfully retrieve an item from
memory despite the face that there are no close connections between retrieval cues
, and the target. Multiple links that radiate outward toward other nodes - a high d
egree
of fan (eg water).
● Excessive activation —> solution: implementation of an inhibitory network.
● Reconstructive memory: constitutes a separate process of retrieving items - one
that does not rely on spreading activation and the inherent, automatic characteristics
of the network. Guided search - one governed by intelligence and reasoning (‘What
did you do on your birthday last year?’).
Network Science
Network science: to explore the way in which complex networks operate. A network is
considered as any collection of interconnected and interacting parts. It’s interdisciplinary.
Contemporary network scientists additionally consider networks as dynamical systems that
are doing things. All networks share some universal mechanism of action.
Centrality
Issue of centrality: how a network coordinates information. This can be accomplished
through a “leader” that receives information, evaluates it, and issues commands. Computers,
armies etc are systems of this kind. But the interesting case is how networks without any
such center achieve this kind of coordinated action. This question has particular relevance
for the human mind —> Cartesian theater and the homunculus problem. If we could figure
out the centrality issue, we might also determine the answer to the mystery of
consciousness. In some networks, coordinated global activity happens simply as a function
of spreading activation that disperses throughout the system quickly but which can arise
from any part of it.
Hierarchical Networks and the Brain
Connections in hierarchical networks are organized in different levels.
1. Simple cells: cells in the primary visual cortex that code for oriented line segments
2. Complex cells: cells in the visual system that code for an oriented line segment
moving in a particular direction
3. Hypercomplex cells: cells in the visual system that code for angles (two conjoined
oriented line segments) moving in a particular direction
If we extrapolate up in the hierarchy, we end up with cells in the highest layers that code for
large complex objects (“grandmother cells”). The hierarchy allows the visual system to
employ a “divide-and-conquer” strategy where it breaks down the complex visual image into
microscopic features and then assembles these features into parts and then wholes that can
be recognized. Communication between levels in hierarchies can allow for the resolution of
ambiguity in visual perception. Information in the visual system appears to travel in 2
directions. It goes nog only in a feed-forward direction from the eye to the brain but also in a
feedback direction from higher brain centers to lower centers.
Small-World Networks: We can define a small-world network as any network where one
can get from any single point to any other point in only a small number of steps even though
the total number of elements may be exceedingly large.
Ordered and Random Connections: Random networks are networks where the connections
are entirely local and can, therefore, be both short and long distance. In an o rdered
Influenced by the principles of operation and organization of real-world brains.
Connectionism tries to understand how the mind performs certain kinds of operations via
the construction of an artificial neural network (ANN) - a computer simulation of how
populations of actual neurons perform tasks.
Artificial Neural Networks
Traditional computers are serial processors: perform one computation at a time. The brain,
as well as ANNs, are parallel distributed processors: large numbers of computing units
perform their calculations in parallel. Knowledge-based approach: One conceptualizes the
problem and its solution in terms of symbols and transformations on the symbols (used a lot
in AI). Behavior-based approach: A network is allowed to produce a solution on its own.
This does not involve the use of symbols (ANNs). Representations are inherent in ANNs but
do not exist in them in the form of symbols. They exist in most networks as a pattern of
activation among the network’s elements - distributed representation. Local
representation: in the form of activation in a single node in a network.
Pro: They are capable of learning —> adaptively change their responses over time as they
are presented with new information (but also possible in machines that use symbolic
methods).
Characteristics of ANNs:
● Only exist as software simulations that are run on a computer
● Each neuron is represented as a node, and the connections between nodes are
represented as links.
● Signal node: activation value —> runs along the link that connects it to another
node(s)
● Input > threshold value —> fire
● Links have weights: specify the strength of a link. Higher value, higher weight.
Early conceptions of Neural Networks
First researchers to propose how biological networks might function: McCulloch and Pitts,
1943. They assumed each neuron had a binary output, it could either send out a signal or
not send out a signal. Donald O. Hebb (1949) was the first to propose how changes among
neurons might explain learning —> Hebb rule: when one cell repeatedly activates another,
the strength of the connection between two cells is increased. He defined 2 types of cell
groupings:
1. Cell assembly: a small group of neurons that repeatedly stimulate one another
2. Phase sequence: a group of connected cell assemblies that fire synchronously or
nearly synchronously
Rosenblatt introduced in 1958 the perceptron: neural nets designed to detect and recognize
patterned information about the world, store this information, and use it in some fashion.
They also learn from experience: can modify their connection strengths by comparing their
actual output with a desired output called the teacher.
Back Propagation and Convergent Dynamics
,Three layer network:
1. Input layer - a representation of the stimulus is presented
2. Hidden layer - feeds activation energy to an output layer
3. Output layer - generates a representation of the response
Error signal: the difference between the actual and the desired outputs. The network uses
the error signal to modify the weights of the links. The kind of training based on error
feedback is called the generalized delta rule or the back-propagation learning model.
NETtalk Is an ANN designed to read written English. Presented written letters —>
pronounces them —> fed to a speech synthesizer for the production of the sounds. System
consists of 3 layers.
Connectionist Approach:
Pro:
● The similarity between network models and real-life neural networks: b iological
plausibility.
○ Artificial Networks share general structural and functional correlates with
biological networks
○ Artificial networks are capable of learning
○ Artificial networks react to damage in the same way that human brains do:
neural networks demonstrate graceful degradation - gradual decrease in
performance with increased damage to the network. Small amounts of
damage —> small reductions in performance.
● Displays interference (2 sets of information are similar in content and interfere with
each other) and generalization (represented by the ability to apply a learned rule to
a novel situation)
Con:
● Biological plausibility should also be viewed as problematic
○ Real neurons are massively parallel, it is not yet possible to simulate parallel
processing of this magnitude.
○ Most networks show a convergent dynamics approach, the activity of such a
network eventually dies down and reaches a stable state. This is not the case
for brain activity. Real neural networks are oscillatory and chaotic.
● Networks may have inadequate learning rules
○ Stability-plasticity dilemma: states that a network should be plastic enough
to store novel input patterns; at the same time, it should be stable enough to
prevent previously encoded patterns form being erased. The fact that ANNs
show evidence of being caught in this dilemma is useful because it may offer
some insights into human interference.
○ Catastrophic interference: occurs in instances in which a network has
learned to recognize a set of patterns and then is called on to learn a new set.
The newly learned patterns suddenly and completely erase the network’s
memory of the original patterns.
○ In supervised networks, a “teacher” is necessary for the network to learn.
But where does this teacher come from?
,Semantic Networks
In semantic networks each node has a specific meaning and, therefore, employs local
representation of concepts. Has been adopted by cognitive psychologists as a way to
explain the organization and retrieval of information in long-term memory.
Characteristics of Semantic Networks:
● A node’s activity can spread outward along links to activate other nodes, these nodes
can then activate still others: spreading activation. Is thought to underlie retrieval of
information from long-term memory. Alternate associations that facilitate recall are
also called retrieval cues.
● The distance between two nodes is determined by their degree of relatedness.
● Priming: the processing of a stimulus is facilitated by the network’s prior exposure to
a related stimulus.
Hierarchical Semantic Network
Study by Collins and Quillian suggests that semantic networks may have a h ierarchical
organization, with different levels representing concepts ranging from the most abstract
down to the most concrete. They used a sentence verification task.
1. Superordinate category: animals —> eat food, breathe
2. Ordinate categories: birds, cats —> can fly, purr
3. Subordinate categories: Canary, Alleycat —> can sing, is yellow
A canary is an animal —> longer reaction time than ‘A canary is a bird/a canary’
“isa” and “has a” link, bird “isa” animal, bird “hasa" feathers
Con:
● Concepts may be represented by prototypes that represent generic or idealized
versions of those concepts.
● Principle of cognitive economy: nodes should not have to be coded for more times
than is necessary. Seems to work better in principle than in reality.
Propositional Semantic Networks
ACT* is a hybrid model: it specifies how multiple memory systems interact and how explicit
knowledge is represented. A proposition is the smallest unit of knowledge that can be
verified. Propositional networks allow for a greater variety of relationships among concepts.
An agent link specifies the subject of the proposition, an object link denotes the object or
thing to which the action is directed. The relation link characterizes the relation between the
agent and the object. Anderson’s ACT* model can also account for the specific memories
each of us has as part of our experience. His model does this via its creation of 2 classes of
nodes: type node; corresponds to an entire category (‘dogs’), t oken nodes; correspond to
specific instances or specific items within a category (“Fido”).
Semantic Networks Evaluation:
Con:
● T.O.T. phenomenon: ‘tip of the tongue’. Semantic Networks cannot easily explain
these sort of retrieval blocks.
● The opposite; the situation in which we can successfully retrieve an item from
memory despite the face that there are no close connections between retrieval cues
, and the target. Multiple links that radiate outward toward other nodes - a high d
egree
of fan (eg water).
● Excessive activation —> solution: implementation of an inhibitory network.
● Reconstructive memory: constitutes a separate process of retrieving items - one
that does not rely on spreading activation and the inherent, automatic characteristics
of the network. Guided search - one governed by intelligence and reasoning (‘What
did you do on your birthday last year?’).
Network Science
Network science: to explore the way in which complex networks operate. A network is
considered as any collection of interconnected and interacting parts. It’s interdisciplinary.
Contemporary network scientists additionally consider networks as dynamical systems that
are doing things. All networks share some universal mechanism of action.
Centrality
Issue of centrality: how a network coordinates information. This can be accomplished
through a “leader” that receives information, evaluates it, and issues commands. Computers,
armies etc are systems of this kind. But the interesting case is how networks without any
such center achieve this kind of coordinated action. This question has particular relevance
for the human mind —> Cartesian theater and the homunculus problem. If we could figure
out the centrality issue, we might also determine the answer to the mystery of
consciousness. In some networks, coordinated global activity happens simply as a function
of spreading activation that disperses throughout the system quickly but which can arise
from any part of it.
Hierarchical Networks and the Brain
Connections in hierarchical networks are organized in different levels.
1. Simple cells: cells in the primary visual cortex that code for oriented line segments
2. Complex cells: cells in the visual system that code for an oriented line segment
moving in a particular direction
3. Hypercomplex cells: cells in the visual system that code for angles (two conjoined
oriented line segments) moving in a particular direction
If we extrapolate up in the hierarchy, we end up with cells in the highest layers that code for
large complex objects (“grandmother cells”). The hierarchy allows the visual system to
employ a “divide-and-conquer” strategy where it breaks down the complex visual image into
microscopic features and then assembles these features into parts and then wholes that can
be recognized. Communication between levels in hierarchies can allow for the resolution of
ambiguity in visual perception. Information in the visual system appears to travel in 2
directions. It goes nog only in a feed-forward direction from the eye to the brain but also in a
feedback direction from higher brain centers to lower centers.
Small-World Networks: We can define a small-world network as any network where one
can get from any single point to any other point in only a small number of steps even though
the total number of elements may be exceedingly large.
Ordered and Random Connections: Random networks are networks where the connections
are entirely local and can, therefore, be both short and long distance. In an o rdered
