Polytopes as vehicles of informational content in feedforward neural networks

Localizing content in neural networks provides a bridge to understanding the way in which the brain stores and processes information. In this paper, I propose the existence of polytopes in the state space of the hidden layer of feedforward neural networks as vehicles of content. I analyze these geometrical structures from an information-theoretic point of view, invoking mutual information to help define the content stored within them. I establish how this proposal addresses the problem of misclassification and provide a novel solution to the disjunction problem, which hinges on the precise nature of the causal-informational framework for content advocated herein.     

The processing of verbs and nouns in neural networks: insights from synthetic brain imaging

The paper presents a computational model of language in which linguistic abilities evolve in organisms that interact with an environment. Each individual's behavior is controlled by a neural network and we study the consequences in the network's internal functional organization of learning to process different classes of words. Agents are selected for reproduction according to their ability to manipulate objects and to understand nouns (objects' names) and verbs (manipulation tasks). The weights of the agents' neural networks are evolved using a genetic algorithm. Synthetic brain imaging techniques are then used to examine the functional organization of the neural networks. Results show that nouns produce more integrated neural activity in the sensory-processing hidden layer, while verbs produce more integrated synaptic activity in the layer where sensory information is integrated with proprioceptive input. Such findings are qualitatively compared with human brain imaging data that indicate that nouns activate more the posterior areas of the brain related to sensory and associative processing, while verbs activate more the anterior motor areas.     

Theory and scope of exact representation extraction from feed-forward networks

An algorithm to extract representations from feed-forward threshold networks is outlined. The representation is based on polytopic decision regions in the input space – and is exact not an approximation. Using this exact representation we explore scope questions, such as when and where do networks form artifacts, or what can we tell about network generalization from its representation. The exact nature of the algorithm also lends itself to theoretical questions about representation extraction in general, such as what is the relationship between factors such as input dimensionality, number of hidden units, number of hidden layers, and how the network output is interpreted to the potential complexity of the network’s function.     

How Do Artificial Neural Networks Classify Musical Triads? A Case Study in Eluding Bonini's Paradox

How might artificial neural networks (ANNs) inform cognitive science? Often cognitive scientists use ANNs but do not examine their internal structures. In this paper, we use ANNs to explore how cognition might represent musical properties. We train ANNs to classify musical chords, and we interpret network structure to determine what representations ANNs discover and use. We find connection weights between input units and hidden units can be described using Fourier phase spaces, a representation studied in musical set theory. We find the total signal coming through these weighted connection weights is a measure of the similarity between two Fourier structures: the structure of the hidden unit's weights and the structure of the stimulus. This is surprising because neither of these Fourier structures is computed by the hidden unit. We then show how output units use such similarity measures to classify chords. However, we also find different types of units—units that use different activation functions—use this similarity measure very differently. This result, combined with other findings, indicates that while our networks are related to the Fourier analysis of musical sets, they do not perform Fourier analyses of the kind usually described in musical set theory. Our results show Fourier representations of music are not limited to musical set theory. Our results also suggest how cognitive psychologists might explore Fourier representations in musical cognition. Critically, such theoretical and empirical implications require researchers to understand how network structure converts stimuli into responses.     

An investigation of trained neural networks from a neurophysiological perspective

The application of theoretical neural networks to preprocessed images was investigated with the aim of developing a computational recognition system. The neural networks were trained by means of a back-propagation algorithm, to respond selectively to computer-generated bars and edges. The receptive fields of the trained networks were then mapped, in terms of both their synaptic weights and their responses to spot stimuli. There was a direct relationship between the pattern of weights on the inputs to the hidden units (the units in the intermediate layer between the input and the output units), and their receptive field as mapped by spot stimuli. This relationship was not sustained at the level of the output units in that their spot-mapped responses failed to correspond either with the weights of the connections from the hidden units to the output units, or with a qualitative analysis of the networks. Part of this discrepancy may be ascribed to the output function used in the back-propagation algorithm.     

Combining predictors for multi-layer architecture of adaptive fuzzy inference system

To solve problems with a Sugeno adaptive fuzzy neural network using training data, it is necessary to select the appropriate combination of input characteristics of the sub-adaptive neuro-fuzzy inference system (ANFIS) and to determine the appropriate topology. The multi-layer architecture of a sub-ANFIS (MLA-ANFIS) is a good model for prediction problems and solves them modularity. Since, the combination of several predictors is the current focus in the construction of hybrid intelligent systems; we created many solutions to combine machine learning methods, namely ANFIS, support vector machine (SVM), deep neural network (DNN), naive Bayes (NB), linear regression (LR), extreme learning machine (ELM), and decision tree (DT) mixed predictors, and ensemble bootstrap aggregation based on MLA-ANFIS in order to discover the optimal model of combined predictors based on the MLA-ANFIS with a combination of input features entered in the MLA-ANFIS. We implemented our approaches on 365-day concrete compressive strength, thoracic surgery, fertility diagnosis, breast, energy, and glass identification datasets from UCI. The experimental results prove that the combining predictors for the MLA-ANFIS show performance improvements compared to the pure MLA-ANFIS method.     

Particle swarm optimised analysis of investment decision

A portfolio forecasting model based on particle swarm optimization (PSO) algorithm with automatic factor scaling is proposed in this Article to effectively improve the accuracy of related analysis model in portfolio application. Firstly, the portfolio problem is analyzed and a hybrid constraint portfolio model is obtained by improving portfolio model with consideration of general portfolio model and combination of market value constraint and upper bound constraint according to Markowitz's theory. Secondly, PSO algorithm is introduced during analysis on portfolio model and solution is found with the hybrid constraint portfolio model of PSO on portfolio. In addition, in order to further improve the performance of PSO in model solution, automatic factor scaling is used for adaptive learning on parameters associated with PSO, to improve convergence of the algorithm. At last, simulation experiments show that the algorithm proposed can obtain a more ideal investment portfolio scheme, thereby reducing investment risks and obtaining greater investment returns.     

Multi-layer architecture for adaptive fuzzy inference system with a large number of input features

The Sugeno adaptive fuzzy neural network using training data is a good approximation to model different systems. The large number of adaptive neuro-fuzzy inference system (ANFIS) input features is a major challenge in using ANFIS and is not applicable with increased parameters. We present a solution for many input features solving modular problems; we created a multi-layer architecture of SUB-ANFIS (MLA-ANFIS) for this purpose. Different topologies were created with various combinations of multiple input features, and an error indicator was calculated for each combination of topologies. Finally, the best topology was chosen among the states with the highest possible performance. We implemented a multi-layered approach based on 365-day concrete compressive strength data with eight input features and the optimized MLA-ANFIS topology (5-3-1) for this purpose from different ANFIS topologies and neural networks. Finally, the results from five other datasets prove the impact of the proposed MLA-ANFIS approach compared to the neural network method.     

A pseudo-dynamic search ant colony optimization algorithm with improved negative feedback mechanism

To solve low convergence precision and slow convergence speed, a pseudo-dynamic search ant colony optimization algorithm with improved negative feedback mechanism (PACON) is proposed. Firstly, the algorithm introduces an angle in the pheromone transfer rule. Through the rule for calculating the angle, multiple cities with smaller angles are also included in the next candidate city list. It affects the probability of city selection and enhances the algorithm’ performance to avoid local optimization. Secondly, the algorithm updates the pheromone concentrations on the worst and optimal path simultaneously, and enhances the weights of the pheromone concentrations on the optimal path. It improves the convergence speed of the algorithm. Based on experiments adopting TSPLIB data sets, the results demonstrate the improved algorithm improves the convergence accuracy by at least 1.26% and increases the convergence speed by at least 9.5%, both on large-scale and small-scale urban data. The novel algorithm will improve convergence precision and speed better.     

Lexical effects on compensation for coarticulation: the ghost of Christmash past

The question of when and how bottom‐up input is integrated with top‐down knowledge has been debated extensively within cognition and perception, and particularly within language processing. A long running debate about the architecture of the spoken‐word recognition system has centered on the locus of lexical effects on phonemic processing: does lexical knowledge influence phoneme perception through feedback, or post‐perceptually in a purely feedforward system? Elman and McClelland (1988) reported that lexically restored ambiguous phonemes influenced the perception of the following phoneme, supporting models with feedback from lexical to phonemic representations. Subsequently, several authors have argued that these results can be fully accounted for by diphone transitional probabilities in a feedforward system (Cairns et al., 1995; Pitt & McQueen, 1998). We report results strongly favoring the original lexical feedback explanation: lexical effects were present even when transitional probability biases were opposite to those of lexical biases.     

Phonology, reading acquisition, and dyslexia: insights from connectionist models.

The development of reading skill and bases of developmental dyslexia were explored using connectionist models. Four issues were examined: the acquisition of phonological knowledge prior to reading, how this knowledge facilitates learning to read, phonological and nonphonological bases of dyslexia, and effects of literacy on phonological representation. Compared with simple feedforward networks, representing phonological knowledge in an attractor network yielded improved learning and generalization. Phonological and surface forms of developmental dyslexia, which are usually attributed to impairments in distinct lexical and nonlexical processing "routes," were derived from different types of damage to the network. The results provide a computationally explicit account of many aspects of reading acquisition using connectionist principles.     

Greater Intermanual Transfer in the Elderly Suggests Age-Related Bilateral Motor Cortex Activation Is Compensatory

ABSTRACT. Hemispheric lateralization of movement control diminishes with age; whether this is compensatory or maladaptive is debated. The authors hypothesized that if compensatory, bilateral activation would lead to greater intermanual transfer in older subjects learning tasks that activate the cortex unilaterally in young adults. They studied 10 young and 14 older subjects, learning a unimanual visuomotor task comprising a feedforward phase, where there is unilateral cortical activation in young adults, and a feedback phase, which activates the cortex bilaterally in both age groups. Increased intermanual transfer was demonstrated in older subjects during feedforward learning, with no difference between groups during feedback learning. This finding is consistent with bilateral cortical activation being compensatory to maintain performance despite declining computational efficiency in neural networks.     

A connectionist model of category learning by individuals with high-functioning autism spectrum disorder

Individuals with autism spectrum disorder (ASD) show atypical patterns of learning and generalization. We explored the possible impacts of autism-related neural abnormalities on perceptual category learning using a neural network model of visual cortical processing. When applied to experiments in which children or adults were trained to classify complex two-dimensional images, the model can account for atypical patterns of perceptual generalization. This is only possible, however, when individual differences in learning are taken into account. In particular, analyses performed with a self-organizing map suggested that individuals with high-functioning ASD show two distinct generalization patterns: one that is comparable to typical patterns, and a second in which there is almost no generalization. The model leads to novel predictions about how individuals will generalize when trained with simplified input sets and can explain why some researchers have failed to detect learning or generalization deficits in prior studies of category learning by individuals with autism. On the basis of these simulations, we propose that deficits in basic neural plasticity mechanisms may be sufficient to account for the atypical patterns of perceptual category learning and generalization associated with autism, but they do not account for why only a subset of individuals with autism would show such deficits. If variations in performance across subgroups reflect heterogeneous neural abnormalities, then future behavioral and neuroimaging studies of individuals with ASD will need to account for such disparities.     

Logic programs and connectionist networks

One facet of the question of integration of Logic and Connectionist Systems, and how these can complement each other, concerns the points of contact, in terms of semantics, between neural networks and logic programs. In this paper, we show that certain semantic operators for propositional logic programs can be computed by feedforward connectionist networks, and that the same semantic operators for first-order normal logic programs can be approximated by feedforward connectionist networks. Turning the networks into recurrent ones allows one also to approximate the models associated with the semantic operators. Our methods depend on a well-known theorem of Funahashi, and necessitate the study of when Funahashi's theorem can be applied, and also the study of what means of approximation are appropriate and significant.     

Connectionist simulation of attitude learning: asymmetries in the acquisition of positive and negative evaluations

Connectionist computer simulation was employed to explore the notion that, if attitudes guide approach and avoidance behaviors, false negative beliefs are likely to remain uncorrected for longer than false positive beliefs. In Study 1, the authors trained a three-layer neural network to discriminate "good" and "bad" inputs distributed across a two-dimensional space. "Full feedback" training, whereby connection weights were modified to reduce error after every trial, resulted in perfect discrimination. "Contingent feedback," whereby connection weights were only updated following outputs representing approach behavior, led to several false negative errors (good inputs misclassified as bad). In Study 2, the network was redesigned to distinguish a system for learning evaluations from a mechanism for selecting actions. Biasing action selection toward approach eliminated the asymmetry between learning of good and bad inputs under contingent feedback. Implications for various attitudinal phenomena and biases in social cognition are discussed.     

Regularized ELM bagging model for Tropical Cyclone Tracks prediction in South China Sea

This paper aims to improve the prediction accuracy of Tropical Cyclone Tracks (TCTs) over the South China Sea (SCS) with 24 h lead time. The model proposed in this paper is a regularized extreme learning machine (ELM) ensemble using bagging. The method which turns the original problem into quadratic programming (QP) problem is proposed in this paper to solve lasso and elastic net problem in ELM. The forecast error of TCTs data set is the distance between real position and forecast position. Compared with the stepwise regression method widely used in TCTs, 8.26 km accuracy improvement is obtained by our model based on the dataset with 70/1680 testing/training records. By contrast, the improvement using this model is 16.49 km based on a smaller dataset with 30/720 testing/training records. Results show that the regularized ELM bagging has a general better generalization capacity on TCTs data set.     

Abstract noun classification: using a neural network to match word context and word meaning

Psychologists have used artificial neural networks for a few decades to simulate perception, language acquisition, and other cognitive processes. This paper discusses the use of artificial neural networks in research on semantics—in particular, in the investigation of abstract noun meanings. It is widely acknowledged that a word’s meaning varies with its contexts of use, but it is a complex task to identify which context elements are relevant to a word’s meaning. The present study illustrates how connectionist networks can be used to examine this problem. A simple feedforward network learned to distinguish among six abstract nouns, on the basis of characteristics of their contexts, in a corpus of randomly selected naturalistic sentences.     

Categorical versus coordinate spatial relations: computational analyses and computer simulations.

Results of 4 sets of neural network simulations support the distinction between categorical and coordinate spatial relations representations: (a) Networks that were split so that different hidden units contributed to each type of judgment performed better than unsplit networks; the reverse was observed when they made 2 coordinate judgments. (b) Both computations were more difficult when finer discriminations were required; this result mirrored findings with human Ss. (c) Networks with large, overlapping "receptive fields" performed the coordinate task better than did networks with small, less overlapping receptive fields, but vice versa for the categorical task; this suggests a possible basis for observed cerebral lateralization of the 2 kinds of processing. (d) The previously observed effect of stimulus contrast on this hemispheric asymmetry could reflect contributions of more neuronal input in high-contrast conditions.     

The biological efficacy of the transfer function

The most popular neural network strategy is back propagation. This strategy initiated general interest in neural networks among researchers. While back propagation can solve nonlinear problems, it is considered to be a poor example of neuron functioning. Recently, Gardner (1993) has made a strong case for a back propagating phenomenon in networks of living neurons. In this paper, we present a few simple computational examples that investigate another component of the typical back propagation network. The effects of varying transfer functions are illustrated along with the resulting variations in possible synaptic weights. Graphic presentations in 3-D space of the relationship between transfer functions and synaptic weights suggest neural analogies of cell-firing rate and network control.