Is it worth generating rules from neural network ensembles?

Although many authors generated comprehensible models from individual networks, much less work has been done in the explanation of ensembles. DIMLP is a special neural network model from which rules are generated at the level of a single network and also at the level of an ensemble of networks. We applied ensembles of 25 DIMLP networks to several datasets of the public domain and a classification problem related to post-translational modifications of proteins. For the classification problems of the public domain, the average predictive accuracy of rulesets extracted from ensembles of neural networks was significantly better than the average predictive accuracy of rulesets generated from ensembles of decision trees. By varying the architectures of DIMLP networks we found that the average predictive accuracy of rules, as well as their complexity were quite stable. The comparison to other rule extraction techniques applied to neural networks showed that rules generated from DIMLP ensembles gave very good results. In the last problem related to bioinformatics, the best result obtained by ensembles of DIMLP networks was also significantly better than the best result obtained by ensembles of decision trees. Thus, although neural networks take much longer to train than decision trees and also rules are generated at a greater computational cost (however, still polynomial), at least for several classification problems it was worth using neural network ensembles, as extracted rules were more accurate, on average. The DIMLP software is available for PC-Linux under http://us.expasy.org/people/Guido.Bologna.html.     

Similarity and rules United: similarity- and rule-based processing in a single neural network

A central controversy in cognitive science concerns the roles of rules versus similarity. To gain some leverage on this problem, we propose that rule- versus similarity-based processes can be characterized as extremes in a multidimensional space that is composed of at least two dimensions: the number of features (Pothos, 2005) and the physical presence of features. The transition of similarity- to rule-based processing is conceptualized as a transition in this space. To illustrate this, we show how a neural network model uses input features (and in this sense produces similarity-based responses) when it has a low learning rate or in the early phases of training, but it switches to using self-generated, more abstract features (and in this sense produces rule-based responses) when it has a higher learning rate or is in the later phases of training. Relations with categorization and the psychology of learning are pointed out.     

Transitive choices by a simple, fully connected, backpropagation neural network: implications for the comparative study of transitive inference.

In search of the minimal requirements for transitive reasoning, a simple neural network was trained and tested on the non-verbal version of the conventional "five-term-series task" – a paradigm used with human adults, children and a variety of non-human species. The transitive performance of the network was analogous in several aspects to that reported for children and animals. The three effects usually associated with transitive choices i.e. "symbolic distance", "lexical marking" and "end-anchor", were also clearly shown by the neural network. In a second experiment, where the training conditions were manipulated, the network failed to match the behavioural pattern reported for human adults in the test following an ordered presentation of the premises. However, it mimicked young children's performance when tested with a novel comparison term. Although we do not intend to suggest a new model of transitive inference, we conclude, in line with other authors, that a simple error-correcting rule can generate transitive behaviour similar to the choice pattern of children and animals in the binary form of the five-term-series task without requiring high-order logical or paralogical abilities. The analysis of the training history and of the final internal structure of the network reveals the associative strategy employed. However, our results indicate that the scope of the associative strategy used by the network might be limited. The extent to which the conventional five-term-series task, in absence of appropriate manipulations of training and testing conditions, is suitable to detect cognitive differences across species is also discussed on the basis of our results. Accepted after revision: 29 May 2001 Electronic Publication     

Efficiency investigation from shallow to deep neural network techniques in human activity recognition

In the last years, several researchers measured different recognition rates with different artificial neural network (ANN) techniques on public data sets in the human activity recognition (HAR) problem. However an overall investigation does not exist in the literature and the efficiency of complex and deeper ANNs over shallow networks is not clear. The purpose of this paper is to investigate the recognition rate and time requirement of different kinds of ANN approaches in HAR. This work examines the performance of shallow ANN architectures with different hyper-parameters, ANN ensembles, binary ANN classifier groups, and convolutional neural networks on two public databases. Although the popularity of binary classifiers, classifier ensembles and deep learning have been significantly increasing, this study shows that shallow ANNs with appropriate hyper-parameters in combination with extracted features can reach similar or higher recognition rate in less time than other artificial neural network methods in HAR. With a well-tuned ANN we outperformed all previous results on two public databases. Consequently, instead of the more complex ANN techniques, the usage of simple ANN with two or three layers can be an appropriate choice for activity recognition.     

Intelligence is differentially related to neural effort in the task-positive and the task-negative brain network

Previous studies on individual differences in intelligence and brain activation during cognitive processing focused on brain regions where activation increases with task demands (task-positive network, TPN). Our study additionally considers brain regions where activation decreases with task demands (task-negative network, TNN) and compares effects of intelligence on neural effort in the TPN and the TNN. In a sample of 52 healthy subjects, functional magnetic resonance imaging was used to determine changes in neural effort associated with the processing of a working memory task. The task comprised three conditions of increasing difficulty: (a) maintenance, (b) manipulation, and (c) updating of a four-letter memory set. Neural effort was defined as signal increase in the TPN and signal decrease in the TNN, respectively. In both functional networks, TPN and TNN, neural effort increased with task difficulty. However, intelligence, as assessed with Raven's Matrices, was differentially associated with neural effort in the TPN and TNN. In the TPN, we observed a positive association, while we observed a negative association in the TNN. In terms of neural efficiency (i.e., task performance in relation to neural effort expended on task processing), more intelligent subjects (as compared to less intelligent subjects) displayed lower neural efficiency in the TPN, while they displayed higher neural efficiency in the TNN. The results illustrate the importance of differentiating between TPN and TNN when interpreting correlations between intelligence and fMRI measures of brain activation. Importantly, this implies the risk of misinterpreting whole brain correlations when ignoring the functional differences between TPN and TNN.     

A nonlinear dynamic artificial neural network model of memory

Nonlinearity and dynamics in psychology are found in various domains such as neuroscience, cognitive science, human development, etc. However, the models that have been proposed are mostly of a computational nature and ignore dynamics. In those models that do include dynamic properties, only fixed points are used to store and retrieve information, leaving many principles of nonlinear dynamic systems (NDS) aside; for instance, chaos is often perceived as a nuisance. This paper considers a nonlinear dynamic artificial neural network (NDANN) that implements NDS principles while also complying with general neuroscience constraints. After a theoretical presentation, simulation results will show that the model can exhibit multi-valued, fixed-point, region-constrained attractors and aperiodic (including chaotic) behaviors. Because the capabilities of NDANN include the modeling of spatiotemporal chaotic activities, it may be an efficient tool to help bridge the gap between biological memory neural models and behavioral memory models.     

Testing neural network models of personality

Attempts to develop neural network models of personality have not generally used empirical data for training and validating the models. Two illustrations are provided which demonstrate the incorporation of empirical data into the modeling of behavioral responses to situations varying in closeness and hierarchical role relationships. An event-contingent recording procedure is utilized to obtain data from the same participant in multiple events for multiple situations. This data is then used in the training and validation of the neural networks. The first illustration models dominant and submissive behaviors in response to situations varying in social role status. The second illustration models agreeable and quarrelsome behaviors in response to situations varying in closeness and gender of the interaction partner. The predictions from both neural network models are consistent with previous research.     

Human age classification using facial skin aging features and artificial neural network

In this paper a novel method based on facial skin aging features and Artificial Neural Network (ANN) is proposed to classify the human face images into four age groups. The facial skin aging features are extracted by using Local Gabor Binary Pattern Histogram (LGBPH) and wrinkle analysis. The ANN classifier is designed by using two layer feedforward backpropagation neural networks. The proposed age classification framework is trained and tested with face images from PAL face database and shown considerable improvement in the age classification accuracy up to 94.17% and 93.75% for male and female respectively.     

Enterprise credit risk evaluation based on neural network algorithm

To explore the enterprise credit risk evaluation, the application effect of several common neural network models in Chinese small and medium-sized enterprise data sets was compared and the optimal parameters for each model were determined. In addition, the classification accuracy and the applicability of the model were compared, and finally the common problem of optimization neural network algorithm based on population was solved: need to determine the dimensions in advance. The experimental results showed that the probabilistic neural network (PNN) had the minimum error rate and second types of errors, while the PNN model had the highest AUC value and was robust. To sum up, the algorithm makes some contributions to solve the financing problem of small and medium-sized enterprises in China.     

The neural network model of organizational identification

This paper proposes the Neural Network Model of Organizational Identification; the model depicts organizational identification as an associative link within an organization member’s social knowledge structure of self as it relates to a focal organization. Within this knowledge structure, organization identification connects self to organization via an attribute sub-network that includes self-concept and organization identity and via a valance sub-network that includes organization based self-esteem and attitudinal commitment. This model draws on the principles of balance-congruity, imbalance dissonance, and differentiation [Greenwald, A. G., Banaji, M. R., Rudman, L. A., Farnham, S. D., Nosek, B. A., & Mellott, D. S. (2002). A unified theory of implicit attitudes, stereotypes, self-esteem, and self-concept. Psychological Review, 109, 3–25.] to predict relationships between these organizational constructs. The Neural Network Model of Organizational Identification is parsimonious yet it effectively integrates and synthesizes the burgeoning literature on organizational identification. By operating at a neural network level of analysis, the model departs substantially from existing organization models by (1) specifying unique construct definitions; (2) offering an alternative perspective of the affective/cognitive dimensions and interrelationships; (3) introducing the concept of implicit cognition to the literature on organizational identification, which makes apparent problems with current measures; and (4) explaining phenomena not explained in existing models. This perspective adds precision and reveals that organizational identification is interconnected within a reciprocal network of mutual causality.     

Learning and development in neural networks--the importance of prior experience

Infants can discriminate between familiar and unfamiliar grammatical patterns expressed in a vocabulary that is distinct from that used earlier during familiarization (Cognition 70(2) (1999) 109; Science 283 (1999) 77). Various models have captured the data, although each required that discrimination be distinct, in terms of the computational process, from familiarization. This article describes a simple recurrent network (SRN), equipped only with the assumption that it should predict what comes next, which models the data without distinguishing between familiarization and discrimination. To accomplish this, the SRN requires pre-training on a range of sequences instantiating different structures and different vocabulary items to those used subsequently during familiarization and test. Pre-training enables the network to avoid replacing structure acquired during familiarization with structure experienced at test. An equivalent enabling condition may underpin infants' resistance to catastrophic interference between the different structures and vocabulary items to which they are exposed.     

Linking patterns of infant eye movements to a neural network model of the ventral stream using representational similarity analysis

Little is known about the development of higher-level areas of visual cortex during infancy, and even less is known about how the development of visually guided behavior is related to the different levels of the cortical processing hierarchy. As a first step toward filling these gaps, we used representational similarity analysis (RSA) to assess links between gaze patterns and a neural network model that captures key properties of the ventral visual processing stream. We recorded the eye movements of 4- to 12-month-old infants (N = 54) as they viewed photographs of scenes. For each infant, we calculated the similarity of the gaze patterns for each pair of photographs. We also analyzed the images using a convolutional neural network model in which the successive layers correspond approximately to the sequence of areas along the ventral stream. For each layer of the network, we calculated the similarity of the activation patterns for each pair of photographs, which was then compared with the infant gaze data. We found that the network layers corresponding to lower-level areas of visual cortex accounted for gaze patterns better in younger infants than in older infants, whereas the network layers corresponding to higher-level areas of visual cortex accounted for gaze patterns better in older infants than in younger infants. Thus, between 4 and 12 months, gaze becomes increasingly controlled by more abstract, higher-level representations. These results also demonstrate the feasibility of using RSA to link infant gaze behavior to neural network models. A video abstract of this article can be viewed at https://youtu.be/K5mF2Rw98Is     

Cost minimization during simulated evolution of paired neural networks leads to asymmetries and specialization

Motivated by specialization (lateralization) that occurs in corresponding left and right regions of the cerebral cortex, several past computational models have studied conditions under which functional specialization can arise during learning due to underlying asymmetries in paired neural networks. However, these past studies have not addressed the basic issue of how such underlying asymmetries arise in the first place. As an initial step in addressing this issue, we investigated the hypothesis that underlying asymmetries will appear in paired neural networks during a simulated evolutionary process when fitness is based not only on maximizing performance, but also on minimizing various ‘costs’ such as energy consumption, neural connection weights, and response times. Simulated evolution under these conditions consistently produced networks with left–right asymmetries in region size, excitability and plasticity. These underlying asymmetries were often synergistic, leading to subsequent functional lateralization during network training. While our computational models are too simple for these results to be directly extrapolated to real nervous systems, they provide support for the hypothesis that brain asymmetries and lateralization in biological nervous systems may be a consequence of cost minimization present during evolution, and are the first computational demonstration of emergent population lateralization.     

An evolutionary system for neural logic networks using genetic programming and indirect encoding

Nowadays, intelligent connectionist systems such as artificial neural networks have been proved very powerful in a wide area of applications. Consequently, the ability to interpret their structure was always a desirable feature for experts. In this field, the neural logic networks (NLN) by their definition are able to represent complex human logic and provide knowledge discovery. However, under contemporary methodologies, the training of these networks may often result in non-comprehensible or poorly designed structures. In this work, we propose an evolutionary system that uses current advances in genetic programming that overcome these drawbacks and produces neural logic networks that can be arbitrarily connected and are easily interpretable into expert rules. To accomplish this task, we guide the genetic programming process using a context-free grammar and we encode indirectly the neural logic networks into the genetic programming individuals. We test the proposed system in two problems of medical diagnosis. Our results are examined both in terms of the solution interpretability that can lead in knowledge discovery, and in terms of the achieved accuracy. We draw conclusions about the effectiveness of the system and we propose further research directions.     

Using spotted hyena optimizer for training feedforward neural networks

Spotted hyena optimizer (SHO) is a novel metaheuristic optimization algorithm based on the behavior of spotted hyena and their collaborative behavior in nature. In this paper, we design a spotted hyena optimizer for training feedforward neural network (FNN), which is regarded as a challenging task since it is easy to fall into local optima. Our objective is to apply metaheuristic optimization algorithm to tackle this problem better than the mathematical and deterministic methods. In order to confirm that using SHO to train FNN is more effective, five classification datasets and three function-approximations are applied to benchmark the performance of the proposed method. The experimental results show that the proposed SHO algorithm for optimization FNN has the best comprehensive performance and has more outstanding performance than other the state-of-the-art metaheuristic algorithms in terms of the performance measures.     

Investigation of psycho-physiological interactions between patient and therapist during a psychodynamic therapy and their relation to speech using in terms of entropy analysis using a neural network approach

In the present contribution we investigate in an exemplary single-case study the behavior of psycho-physiological variables in psychotherapy sessions. The values are measured continously during a single session at the same time for both patient and therapist. The analysis of the data is done using an artificial neural network approach for non-linear principal component analysis and faithful data representation/visualization and compression required for subsequent process analysis. The used network (growing self-organizing map, GSOM) thereby uses a kernel smoothing for improved data density estimation. In this way, we are able to generate an entropy model of psycho-physiological variability detecting emotionally instable phases during the therapy process. We relate our finding to results obtained by speech analysis of the therapy sessions according to the cycle model invented by Mergenthaler. Thus, we get preliminary suggestions how psycho-physiological reactions are related to the therapeutic process.     

Application of improved BP neural network based on e-commerce supply chain network data in the forecast of aquatic product export volume

Aiming at the existing problems in the production and export scale prediction of aquaculture, a model of yield prediction based on BP Neural network algorithm is proposed, and a set of algorithms is proposed to optimize BP neural network (BPNN). Based on the traditional BP neural network, it is easy to get into the local optimal problem due to the long training time of the model. By using the simple Johnson algorithm, the dimensionality of the input neuron is reduced, and then the hidden layer neural network is determined by this method. At the same time, the data mining method is used to filter the Data.Particle swarm optimization algorithm is used to optimize the parameters. At the same time, based on the domestic e-commerce Sales network data, the results show that the average square root error of the model is less than the traditional BP neural network and the learning efficiency is higher than the traditional BP neural network. The results show that the model has a great advantage in building up a large number of historical data, and it can shorten the modeling time and get good prediction result by combining the sales data of e-commerce. It provides a new feasible method for the export prediction of aquatic products.     

The AHA! experience: creativity through emergent binding in neural networks

Many kinds of creativity result from combination of mental representations. This paper provides a computational account of how creative thinking can arise from combining neural patterns into ones that are potentially novel and useful. We defend the hypothesis that such combinations arise from mechanisms that bind together neural activity by a process of convolution, a mathematical operation that interweaves structures. We describe computer simulations that show the feasibility of using convolution to produce emergent patterns of neural activity that can support cognitive and emotional processes underlying human creativity.     

Critical factors in consumer supermarket shopping behaviour: a neural network approach

This paper studies UK supermarket shopping behaviour, by analysing the antecedent variables of three critical factors: overall levels of customer satisfaction, number of trips to the supermarket, and amount spent. A neural network approach predicts these factors using ten input variables and three hidden nodes. Results show that the most satisfied and high‐spending customers tend to be those who have the income to take full advantage of the choice and quality offered. Other customers are more concerned with prices being reasonable and discounts available, but the satisfaction of these shoppers is also linked with store atmosphere. Copyright © 2001 Henry Stewart Publications.