Acquaintanceship and self/observer agreement in personality judgment

Self/observer agreement on HEXACO-PI-R scale scores was examined as a function of observers’ subjective ratings of acquaintanceship. For each participant (N = 2199), personality self-reports were obtained along with observer reports from a friend. Each factor-level scale displayed a different pattern of upward accuracy (agreement) trends in personality judgment. Self/observer agreement for Extraversion, Emotionality, and Openness was noticeably stronger at lower acquaintanceship than that for Conscientiousness, Agreeableness, and Honesty-Humility. Conscientiousness showed a steep upward accuracy trend across acquaintanceship levels, reaching a level of accuracy comparable to that of Extraversion and Emotionality. Self/observer agreement for Honesty-Humility and Agreeableness showed slower upward trends than that of Conscientiousness. In several cases, facet-level traits within the same broad factor differed in their accuracy trends.     

Shaping data collection congruent with experimental hypotheses

The influence of instructions and feedback from an experimenter on observational recordings of disruptive behavior was evaluated. Four subject-observers recorded four categories of disruptive behavior from videotapes of children in a classroom setting. Two sets of videotapes, labelled "baseline" and "treatment", were matched for rates of disruptive behavior in each category. The observers were told that two target behaviors were expected to decrease and the other two control behaviors were not expected to change during the treatment condition. During observational recording of treatment videotapes, the observers were given positive feedback when they reported decreases in the two target behaviors, and negative feedback when they reported either to change or increases in the two control behaviors. The target behaviors were recorded less frequently by observers, while recordings of control behaviors were unaffected during the treatment phase. These results suggest that contingent feedback to observers should be avoided as a possible source of bias in behavioral recordings.     

Evaluation of self-contained training procedures for the Time-Sample Behavioral Checklist

The comparative effectiveness of two time-limited modes of training observers to code the behavior of clients in residential treatment programs on the Time-Sample Behavioral Checklist (TSBC) was evaluated. The susceptibility of training procedures to consensual observer drift and the predictability of TSBC mastery from trainee characteristics were also examined. Two equated groups of undergraduate students (N=15 each) participated in full-time training on the TSBC and another instrument for 27 days, followed by criterion testingin vivo and on videotapes. One group was trained by experienced personnel using procedures known to be effective but potentially subject to consensual observer drift. The other group was trained using a previously untested set of written and videotape procedures that do not rely on experienced personnel. Comparative effectiveness and observer drift were evaluated by multivariate and univariate ANOVAs on mastery scores reflecting both pattern agreement and level differences between each trainee and criterion codings. The new, more efficient training procedures were found to be as effective as the original training procedures in the degree of mastery achieved by trainees. Original training procedures were found to be resistant to consensual observer drift, with such a phenomenon appearing in only 1 instance of 156 opportunities. The high degree of mastery achieved by trainees during the time-limited training period was comparable to that previously achieved with the original training procedures. No meaningful predictions of coding mastery were found, with only one trainee characteristic being significantly related to one of six mastery criteria. The results document procedures that are both efficient and resistant to invalidity for training observers in the use of multidimensional observational systems, as well as providing guidelines for the development of standardized procedures.This article is based on a thesis submitted to the Graduate College of the University of Illinois at Urbana-Champaign in partial fulfillment of the requirements for the Ph.D. degree in psychology by the first author and under direction of the second author. The third and fourth authors also participated as supervisors. Appreciation is extended to other members of the thesis committee, Fred Kanfer, Ross Parke, and Julian Rappaport, and to John Gottman and W. Robert Nay for their comments and recommendations. This study was partially supported by Public Health Service Grants MH-25464 and MH-14257 from the National Institute of Mental Health and by grants from the Joyce Foundation, the MacArthur Foundation, and the Illinois Department of Mental Health and Developmental Disabilities.     

ASSESSING OBSERVER ACCURACY IN CONTINUOUS RECORDING OF RATE AND DURATION: THREE ALGORITHMS COMPARED

The three algorithms most frequently selected by behavior‐analytic researchers to compute interobserver agreement with continuous recording were used to assess the accuracy of data recorded from video samples on handheld computers by 12 observers. Rate and duration of responding were recorded for three samples each. Data files were compared with criterion records to determine observer accuracy. Block‐by‐block and exact agreement algorithms were susceptible to inflated agreement and accuracy estimates at lower rates and durations. The exact agreement method appeared to be overly stringent for recording responding at higher rates (23.5 responses per minute) and for higher relative duration (72% of session). Time‐window analysis appeared to inflate accuracy assessment at relatively high but not at low response rate and duration (4.8 responses per minute and 8% of session, respectively).     

Observer agreement, credibility, and judgment: some considerations in presenting observer agreement data

Graphical and statistical indices employed to represent observer agreement in interval recording are described as "judgmental aids", stimuli to which the researcher and scientific community must respond when viewing observer agreement data. The advantages and limitations of plotting calibrating observer agreement data and reporting conventional statistical aids are discussed in the context of their utility for researchers and research consumers of applied behavior analysis. It is argued that plotting calibrating observer data is a useful supplement to statistical aids for researchers but is of only limited utility for research consumers. Alternatives to conventional per cent agreement statistics for research consumers include reporting special agreement estimates (e.g., per cent occurrence agreement and nonoccurrence agreement) and correlational statistics (e.g., Kappa and Phi).     

The Effects of Instructions on Mothers’ Ratings of Attention-Deficit/Hyperactivity Disorder Symptoms in Referred Children

Tested whether instructions for how to rate child attention-deficit/hyperactivity disorder (ADHD) symptoms would improve the agreement between mothers’ ratings of symptoms in their children and ratings provided by teachers and objective observers. Sixty-eight mothers of 5 to 12 year old children (53 boys and 15 girls) referred for ADHD assessment were randomly assigned to receive or not receive the instructions. Mothers and teachers rated the children on the SNAP-IV Rating Scale and objective observers rated the children’s behavior during structured tasks. Relations between mother and teacher, and mother and observer ratings were generally stronger for mothers in the Instruction group compared to mothers in the No Instruction group, in some cases significantly stronger. The instructional materials also improved mothers’ knowledge of how to rate ADHD symptoms and reduced some associations between mothers’ ratings and family socioeconomic status. These instructions have the potential to improve clinical assessments of child ADHD symptoms.     

Retention and Recall: Incidental Learning of Visual and Auditory Material

This study examined the effect on observer agreement of switching from a system of overt reliability assessment to two successive systems of covert reliability measurement. A primary purpose was to see whether agreement obtained with the use of covert data checks would improve over time if observers were provided with accurate feedback regarding their level of agreement. Seventeen undergraduate psychology students served as Ss. They observed instructional interactions between preschool children and their teachers. During the two week “overt” check period (Phase I), observers were aware of when their observations were being “checked” by a previously designated reliability assessor. In the subsequent covert phases (weeks 3–7) this information was not available to them during the observational sessions. When covert monitoring was implemented, agreement rates initially dropped significantly below the “overt” measurement phase. Gradually agreement rates improved until, in weeks 6 and 7, they were not significantly different from the initial overt measures. Es should be aware that an overt check on observer agreement may not reflect the true reliability of an observational system. However, when observers are given accurate feedback on their level of agreement, they are able significantly to improve their vigilance and consistency in the use of the observational system.     

Observer reliability as a function of circumstances of assessment

THREE FACTORS CHARACTERISTIC OF EXPERIMENTAL SETTINGS WERE HYPOTHESIZED TO INFLATE ARTIFACTUALLY THE RELIABILITY OF OBSERVATIONAL RECORDINGS: (a) knowledge by observers of when and by whom their reliability is being assessed, (b) the absence of the experimenter or a monitor to prevent cheating, and (c) computation of reliability within- (versus between-) observer group. Three groups of four observers used a standard nine-category observational code for disruptive behavior in recording from videotapes of a classroom for 22 days. Analyses revealed considerable increases in average occurrence reliability as a function of the main effects of each of the experimental factors. The specific increases in reliability associated with each of the 12 combinations of the experimental factors are presented for each category of behavior. The possible role of observer-training procedures and behavioral definitions as determiners of nonartifactual reliability is discussed.     

Horse sense: social status of horses (<Emphasis Type="Italic">Equus caballus</Emphasis>) affects their likelihood of copying other horses’ behavior

Animals that live in stable social groups need to gather information on their own relative position in the group's social hierarchy, by either directly threatening or by challenging others, or indirectly and in a less perilous manner , by observing interactions among others. Indirect inference of dominance relationships has previously been reported from primates, rats, birds, and fish. Here, we show that domestic horses, Equus caballus, are similarly capable of social cognition. Taking advantage of a specific "following behavior" that horses show towards humans in a riding arena, we investigated whether bystander horses adjust their response to an experimenter according to the observed interaction and their own dominance relationship with the horse whose reaction to the experimenter they had observed before. Horses copied the "following behavior" towards an experimenter after watching a dominant horse following but did not follow after observing a subordinate horse or a horse from another social group doing so. The "following behavior," which horses show towards an experimenter, therefore appears to be affected by the demonstrator's behavior and social status relative to the observer.     

Limitations of a Bidirectional Control Procedure for the Investigation of Imitation in Rats: Odour Cues on the Manipulandum

Magazine-trained observer rats confronted a conspecific demonstrator pushing a joystick to the right or to the left for food reward before the observers were given access to the joystick from the position previously occupied by the demonstrator and rewarded for responses in both directions. For half of the observers (group 0), the joystick was in the same position when acted upon by demonstrators and observers; for the other half (group 180) the manipulandum was rotated 180 within its mounting between observation and test. As in previous experiments using this bidirectional control procedure, rats in group 0 showed demonstratorconsistent responding-that is, they pushed the joystick in the same direction, relative to the actor's body, as had their demonstrators. However, group 180 showed a reverse effect: reliable demonstrator-inconsistent responding. These results suggest that attractive odour or taste cues deposited by demonstrators on the side of the joystick contralateral to the direction of responding are sufficient to produce demonstrator-consistent responding in the bidirectional control procedure.     

Hypnotic age regression and moral reasoning

Moral reasoning was assessed with two dilemmas from the Kohlberg Moral Judgment Interview (1976) in 7 adults who were hypnotically age regressed to four ages: 15, 12, 9, and 6 years. Seven adults in a control group under task-motivation instructions were age regressed to the same ages and administered the dilemmas. Age-regressed moral maturity scores for both groups were compared to pretest scores, to data for actual 15-, 12-, 9-, and 6-year-olds from other studies, and to each other. Both groups were able to lower their moral reasoning scores when given age-regression instructions; however, hypnosis subjects not only scored lower than task-motivation subjects but also more closely resembled the levels of reasoning used by children at these ages. The results provide some support for a distinction between age regression through hypnosis and age regression through task-motivation instructions.     

Viewpoint-dependent recognition of familiar faces

The question whether object representations in the human brain are object-centered or viewer-centered has motivated a variety of experiments with divergent results. A key issue concerns the visual recognition of objects seen from novel views. If recognition performance depends on whether a particular view has been seen before, it can be interpreted as evidence for a viewer-centered representation. Earlier experiments used unfamiliar objects to provide the experimenter with complete control over the observer's previous experience with the object. In this study, we tested whether human recognition shows viewpoint dependence for the highly familiar faces of well-known colleagues and for the observer's own face. We found that observers are poorer at recognizing their own profile, whereas there is no difference in response time between frontal and profile views of other faces. This result shows that extensive experience and familiarity with one's own face is not sufficient to produce viewpoint invariance. Our result provides strong evidence for viewer-centered representations in human visual recognition even for highly familiar objects.     

Habituation of orienting responses as a function of “instructional set”

Electrodermal and plethysmographic recordings were obtained from three groups of alcoholic patients. The groups differed with respect to instructions given for response to a series of words spoken by the experimenter. Group A was told to report verbally their first associations for each word presented. Group B received instructions to think of an association for each word without reporting it. Group C was to respond to each word by repeating it after it was spoken by the experimenter. The instructions did not have the predicted differing effects upon rate of EDR habituation. However, the over-all amplitude of EDR was lower for Group B than for Group A or C. We observed very little habituation of the plethysmographic OR as measured by per cent of subjects responding to each word. As with the EDR data, Group B shows less responsivity, i.e., this group produced significantly fewer plethysmographic responses over all words than did either of the other two groups. The relative importance of association and of verbal responding to a stimulus word for eliciting autonomic responses was considered.     

Decision strategies for the two-alternative forced choice reminder paradigm

The decision strategy adopted by an observer in a psychophysical procedure may be established by comparing estimates of sensitivity obtained by using detection-theoretic models based on each of the available decision strategies. Estimates of sensitivity obtained from 10 observers on auditory-level discrimination tasks using the yes/no, two-alternative forced choice (2AFC), and 2AFC with reminder (2AFCR; sometimes called duo-trio) procedures were compared in order to determine the decision strategy used by the judges in the 2AFCR procedure. 2AFCR permits at least three decision strategies: differencing, likelihood ratio, and comparison of distances. The latter strategy has frequently been assumed in the analysis of duo-trio data in the sensory evaluation literature. The present study demonstrates that the comparison-of-distances strategy was not used by 9 of the 10 observers in this auditory experiment. The nature of the instructions given to the observers had no measurable impact on the decision strategy that they adopted.     

Effects of Plotting a Second Observer’s Data on A-B-A-B Graphs When Observer Disagreement is Present

Plotting data from a second observer have been suggested as an alternative to presenting only mean agreement and ranges across sessions. One argument against this method of reporting is that plotting these data would not change analysis decisions by persons analyzing graphs. To empirically test this argument, we asked graduate students to analyze graphs with and without data from a second observer plotted alongside primary data. Results show that when these data are plotted: (a) observers do not change analysis decisions when both or neither paths show a functional relation, (b) observers change analysis decisions when only one path shows a functional relation, though this varied across graphs and participants, and (c) observers who did not change analysis decisions when only one path showed a functional relation often rated their decision with less confidence than when both paths showed a functional relation or when only summary statistics are reported. Findings suggest plotting data from a second observer may be helpful in making analysis decisions for A-B-A-B graphs.     

Ambiguity and the 'mental eye' in pictorial relief

Photographs of scenes do not determine scenes in the sense that infinitely many different scenes could have given rise to any given photograph. In psychophysical experiments, observers have (at least partially) to resolve these ambiguities. The ambiguities also allow them to vary their response within the space of 'veridical' responses. Such variations may well be called 'the beholder's share' since they do not depend causally on the available depth cues. We determined the pictorial relief for four observers, four stimuli, and four different tasks. In all cases we addressed issues of reliability (scatter on repeated trials) and consistency (how well the data can be explained via a smooth surface, any surface). All data were converted to depth maps which allows us to compare the relief from the different operationalisations. As expected, pictorial relief can differ greatly either between observers (same stimulus, same task) or between operationalisations (same observer, same stimulus). However, when we factor out the essential ambiguity, these differences almost completely vanish and excellent agreement over tasks and observers pertains. Thus, observers often resolve the ambiguity in idiosyncratic ways, but mutually agree--even over tasks--in so far as their responses are causally dependent on the depth cues. A change of task often induces a change in 'mental perspective'. In such cases, the observers switch the 'beholder's share', which resolves the essential ambiguity through a change in viewpoint of their 'mental eye'.     

Expert and Target Scoring: Their relation,corresponding test instructions,and their effects on the construct validity of the video‐based social understanding test (VSU)

This study investigated the relation between expert and target scoring of a video‐based social understanding test (VSU) under two different types of instructions (internal and observer). The effects of the scoring methods and instructions on the VSU's construct validity were also examined. A total of 529 pilot applicants completed the VSU (some with internal and some with observer instructions), cognitive ability and knowledge tests, and a personality questionnaire. A subsample (n = 132) completed the VSU again with the other instructions and participated in an assessment center (AC). The two scores were moderately correlated; correlations decreased when the instructions were considered. Neither expert nor target scores showed convergent validity with AC variables; none of the scoring‐instruction combinations showed significant associations with the remaining measures.     

The comparative effectiveness of two modes of observer training on the Staff-Resident Interaction Chronograph

The comparative effectiveness of two time-limited modes of training observers to code activity on the Staff-Resident Interaction Chronograph (SRIC) in residential treatment programs for mentally disabled adults was evaluated. The susceptibility of training procedures for consensual observer drift was also examined, as was the predictability of SRIC mastery from trainee characteristics. Two equated groups of undergraduate student trainees (N=15 each) participated in full-time training for 27 days, followed by two weeks of criterion testing in vivo and on videotapes. One group received training by experience personnel using procedures known to be effective (original method). The other group received training via a previously untested set of written and videotaped procedures that do not rely on experienced personnel (package method). Multivariate and univariate analyses of variance found both methods to be equally effective in the degree of mastery achieved by trainees, without evidence of observer drift. No meaningful predictions of coding mastery were found, but conceptual mastery was predictable from individual characteristics. Differences were obtained for both groups between in vivo versus videotaped criterion tests. The results document procedures that are both efficient and resistant to invalidity for complex observational methodology as well as feasible for standardizing assessment of staff functioning across residential settings.This article is based on a thesis submitted to the Graduate College of the University of Illinois at Urbana-Champaign in partial fulfillment of the requirements for the Ph.D. degree in psychology by the first author under direction of the second author. The third and fourth authors also participated as supervisors. Appreciation is extended to other members of the thesis committee, Fred Kanfer, W. Robert Nay, Julian Rappaport, and James Wardrop, for their comments and recommendations. This study was partially supported by Public Health Service Grants MH-25464 and MH-14257 from the National Institute of Mental Health, and by grants from The Joyce Foundation and the Illinois Department of Mental Health and Developmental Disabilities.     

Judgments of Decision Effectiveness: Actor-Observer Differences in Overconfidence

Subjects playing the role of psychiatrists (actors) engaged in a simulated medical decision-making task in which they attempted to bring the value of a patient indicator variable into a desired range. For each treatment recommended by the actor, both the actor and an observer subject playing the role of a nurse assessed the probability that the treatment would be effective. Both actors and observers were overconfident. Actors were more confident in their treatment recommendations than were observers, but this difference was eliminated when observers were given the opportunity to offer their own alternative recommendation. Under the latter circumstances, actors and observers were equally confident in the actors' decisions but observers were more confident than actors in the observers' decisions. These findings suggest that while control over the outcome of the decision has little influence on actor-observer differences in confidence, feedback regarding this outcome plays a crucial role.