Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability

Peer review is a gatekeeper, the final arbiter of what is valued in academia, but it has been criticized in relation to traditional psychological research criteria of reliability, validity, generalizability, and potential biases. Despite a considerable literature, there is surprisingly little sound peer-review research examining these criteria or strategies for improving the process. This article summarizes the authors' research program with the Australian Research Council, which receives thousands of grant proposals from the social science, humanities, and science disciplines and reviews by assessors from all over the world. Using multilevel cross-classified models, the authors critically evaluated peer reviews of grant applications and potential biases associated with applicants, assessors, and their interaction (e.g., age, gender, university, academic rank, research team composition, nationality, experience). Peer reviews lacked reliability, but the only major systematic bias found involved the inflated, unreliable, and invalid ratings of assessors nominated by the applicants themselves. The authors propose a new approach, the reader system, which they evaluated with psychology and education grant proposals and found to be substantially more reliable and strategically advantageous than traditional peer reviews of grant applications.     

The impact of discredited evidence

How do people revise their beliefs when evidence is discredited? In three studies, mock jurors read simplified criminal cases and then judged the probability that a suspect was guilty on the basis of sequentially presented evidence. Study 1 showed an extension effect: When two items of incriminating evidence were presented, a subsequent discrediting of the second item also lessened belief in the first item, irrespective of whether it was directly related to the discredited item. Study 2 showed that this effect depended on the order of evidence presentation: When the discrediting evidence was presented early, rather than late, in the sequence, there was no extension to unrelated items. Study 3 showed that the extension effect held only when items of evidence were both incriminating or both exonerating, but not when they were mixed. To explain these findings, we draw on coherence-based models of juror reasoning and propose that people group evidence according to its direction with respect to the guilt hypothesis.     

Characterizing the Linguistic Chameleon: Personal and Social Correlates of Linguistic Style Accommodation

Linguistic style accommodation between conversationalists is associated with positive social outcomes. We examine social power and personality as factors driving the occurrence of linguistic style accommodation, and the social outcomes of accommodation. Social power was manipulated to create 144 face‐to‐face dyadic interactions between individuals of high versus low power and 64 neutral power interactions. Particular configurations of personality traits (high self‐monitoring, Machiavellianism and leadership, and low self‐consciousness, impression management and agreeableness), combined with a low‐power role, led to an increased likelihood of linguistic style accommodation. Further, greater accommodation by low‐power individuals positively influenced perceptions of subjective rapport and attractiveness. We propose individual differences interact with social context to influence the conditions under which nonconscious communication accommodation occurs.     

Conceptual and perceptual priming and dissociation in chronic posttraumatic stress disorder

Cognitive models of posttraumatic stress disorder (PTSD) assert that memory processes play a significant role in PTSD (see e.g., Ehlers & Clark, 2000). Intrusive reexperiencing in PTSD has been linked to perceptual processing of trauma-related material with a corresponding hypothesized lack of conceptual processing. In an experimental study that included clinical participants with and without PTSD (N = 50), perceptual priming and conceptual priming for trauma-related, general threat, and neutral words were investigated in a population with chronic trauma-induced complaints as a result of the Troubles in Northern Ireland. The study used a new version of the word-stem completion task (Michael, Ehlers, & Halligan, 2005) and a word-cue association task. It also assessed the role of dissociation in threat processing. Further evidence of enhanced perceptual priming in PTSD for trauma stimuli was found, along with evidence of lack of conceptual priming for such stimuli. Furthermore, this pattern of priming for trauma-related words was associated with PTSD severity, and state dissociation and PTSD group made significant contributions to predicting perceptual priming for trauma words. The findings shed light on the importance of state dissociation in trauma-related information processing and posttraumatic symptoms.     

The functional anatomy of inspection time: a pilot fMRI study

Seven healthy subjects underwent functional magnetic resonance imaging (fMRI) of the brain while performing an inspection time task. Employing a block-type design, the task had three difficulty levels: a control condition, an easy (200 ms stimulus duration), and a more difficult (40 ms) discrimination. Based on group results, there were widespread significant areas of difference in brain activation and deactivation when pairwise comparisons were conducted among the three task conditions. When the difficult condition was compared with the easy condition, there was relative activation in areas of the following brain regions: cingulate gyrus and some frontal and parietal lobe areas. Areas within the following regions showed relative deactivation (greater blood oxygenation level-dependent, BOLD, signal in the easy condition): frontal, temporal, and parietal lobe. There were overlaps between these areas and those found to be active while performing higher cognitive tasks in other functional brain imaging studies. These pilot data encourage future studies of the functional anatomy of inspection time and its relevance to psychometric intelligence.     

Reflectivity, Reflection, and Counter-Education

This article sets forward a new concept of reflection, to be contrasted with more usual reading of the concept for which we use the term `reflectivity'. The contrast is related to a distinction between normalizing education and counter-education. We claim that within the framework of normalizing education there is no room for reflection, but only for reflectivity. In contrast to reflectivity, reflection manifests a struggle of the subject against the effects of power which govern the constitution of her conceptual apparatus, her knowledge, her consciousness and her limitations and possibilities for successful functioning. Reflectivity re-presents the hegemonic realm of self-evidence and the productive violence of social and cultural order. Reflection, by contrast, aims to challenge the supposedly self-evident and the present order of things. Reflection aims at transcendence and represents a moral commitment in respect of the otherness of the Other, which power relations in every realm of self-evidence oblige us to neglect, to destroy or consume. Transcendence is a concrete utopia, and so is the subject in her nonrepressive communication with the Other: they are part and parcel of our present possibilities, sometimes in microscopic arenas, struggled for, and from time to time even realized.     

Neuropsychological Aspects of Multiple Sclerosis

Since S. Rao's [Neuropsychology of Multiple Sclerosis: A Critical Review, A Journal of Clinical and Experimental Neuropsychology, Vol. 85, pp. 503–542] (1986) seminal review, considerable research has been undertaken on the neuropsychological consequences of multiple sclerosis. This review incorporates the research literature of the last decade in presenting an overview of the current state of our knowledge concerning the etiology, course, symptoms, assessment, consequences, and treatment of multiple sclerosis (MS). The concept of subcortical dementia is revisited in light of the most recent literature documenting the neuropsychological deficits in patients with MS. The view that cognitively heterogeneous patient groups may disguise more specific patterns of focal neuropsychological impairment is considered. A critical review of the recent literature is also presented, detailing the degree to which recent research has addressed the areas of research need identified by Rao in 1986. Given recent advances in our knowledge, the need for more attention to be directed toward the evaluation of rehabilitation and psychological intervention is highlighted.