ODAP: a stand-alone program for observational data acquisition

A software program written to collect real-time, observational data is described. The flexible program allows customized behavior codes and observational durations and simultaneously records both timed and counted events. The data are collected by means of single keystrokes, automatically stored to disk with 100th of a second resolution and summarized for each observational session. The program’s database files are dBASE III PLUS compatible and may be browsed, edited, or converted to ASCII files from the program’s main menu. Field testing demonstrated the efficiency and interobserver reliability of the program (for frequency,r=0.81; for durational behaviors,rs=0.89 and 0.96). The software operates on IBM XT/AT/PS 2s and most clones with PC/MS DOS version 2.0 or greater.     

A conceptual and graphical method for converging multisubject behavioral observational data into a single process indicator

Typically, in observational research, each behavior of a subject is assigned a code that represents theoretically relevant and contextually sensitive events. These discrete behavioral acts are then strung together to denote a process. With multiple interactants, the investigator has two choices: ignore subject interdependence and analyze as if the data strings are discrete, or treat the strings as interdependent and create a summary index that represents the combined codes across subjects. This article illustrates a method of combining discrete within-subjects behavioral indicators into behavior-specific domains and then conglomerating the disparate domain strings across subjects in such a way that the aggregate reflects the observed process. This aggregation is represented by a coordination index, expressed as either interactant disparity or coherence. Also, animation and visualization methods are presented that illustrate how social processes can be modeled by using either the coordination index or the separate domain values.     

Estimating the reliability of interview data

A model for a score based on an interview is presented which identifies the effect due to the subject, to the manner in which the interviewer tends to conduct his interviews, to the criteria he tends to use in scoring subjects' responses, to the compromises he tends to adopt between the demands of interviewing and those of scoring, and to chance errors. A suggested experimental design calls for each ofK investigators to interview a different sample ofN subjects, but for all investigators to score each subject. The drawing of inferences when interest is only in theK participants in the reliability study is considered, and a numerical example is given.This work was supported in part by grant DE R01 00793 from the National Institute of Dental Research, and in part by grants MH 08534 and MH 09191 from the National Institute of Mental Health, and forms part of the author's Ph.D. dissertation at Columbia University. The guidance provided by Professor T. W. Anderson is gratefully acknowledged.     

Measuring service learning outcomes: test-retest reliability of four scales

Previous research efforts have developed and validated various scales potentially useful in evaluating service learning outcomes. The developmental efforts reported for the four scales examined in this study did not include the test-retest reliabilities that would provide assurance to service learning researchers of the long-term stability and therefore usefulness of these measures. Summary estimates of 13-wk. test-retest reliabilities for the scales Civic Participation, Self-efficacy Toward Service, Attitude Toward Helping Others, and College Education's Role in Addressing Social Issues provide service learning researchers with evidence of stability of the scales over the typical duration of service learning courses.     

Measuring Shaken Baby Syndrome awareness: preliminary reliability of a caregiver attitudes and beliefs survey

Recent estimates of Shaken Baby Syndrome awareness suggest that approximately half of all American adults have not heard of the often devastating risks of shaking an infant. Using a sample of 288 undergraduate students, we developed a measure of attitudes around infant care practices. A total of 264 community participants completed a revised survey. Between-group comparisons, exploratory factor analyses, and internal consistency tests were employed to determine the directionality and reliability of any scale structure present in the data. A five factor structure fits the data reliably, and each of these factors seems to represent a unique dimension. Implications for using this measure clinically and preventatively are discussed.     

Analyzing sequential relations in observational data: A practical guide

Many observational systems in basic and applied research produce a record of sequences of events over time. Within such observational systems, important information may be found in the frequency of the transitions between events that does not emerge in the typical researcher's focus on absolute event frequency. Indeed, if many behaviors are controlled by closely adjacent preceding events, then substantial prediction and control can be obtained through knowledge and manipulation of causal event transitions. A method for analyzing and testing sequential dependencies between events is proposed as part of an integrated package of computer-based data entry, storage, and analysis procedures. The mathematical portion of these techniques is based on the statistic, kappa,which is applicable to determining whether particular transitions among events differ from chance and whether particular transitions differ significantly across groups of subjects. Low-cost hardware and software to implement the proposed procedures are described.This work has been supported by Grant 1 RO1 HD19245-01A1 from the National Institute of Child Health and Human Development to the two senior authors and by grants of equipment from Commodore Computers, Inc., Koala Corporation, and NEC Telephones, Inc.     

Measuring socioeconomic status: reliability and preliminary validity for different approaches

This study investigated issues related to commonly used socioeconomic status (SES) measures in 140 participants from three cities (Atlanta, Boston, and Toronto) in two countries (United States and Canada). Measures of SES were two from the United States (four-factor Hollingshead scale, Nakao and Treas scale) and one from Canada (Blishen, Carroll, and Moore scale). Reliability was examined both within (interrater agreement) and across (intermeasure agreement) measures. Interrater reliability and classification agreement was high for the total sample (ranger = .86 to .91), as were intermeasure correlations and classification agreement (range r = .81 to .88). The weakest agreement across measures was found when families had one wage earner who was female. Validity data for these SES measures with academic and intellectual measures also were obtained. Some support for a simplified approach to measuring SES was found. Implications of these findings for the use of SES in social and behavioral science research are discussed.     

General estimators for the reliability of qualitative data

We study a proportional reduction in loss (PRL) measure for the reliability of categorical data and consider the general case in which each ofN judges assigns a subject to one ofK categories. This measure has been shown to be equivalent to a measure proposed by Perreault and Leigh for a special case when there are two equally competent judges, and the correct category has a uniform prior distribution. We consider a general framework where the correct category is assumed to have an arbitrary prior distribution, and where classification probabilities vary by correct category, judge, and category of classification. In this setting, we consider PRL reliability measures based on two estimators of the correct category—the empirical Bayes estimator and an estimator based on the judges' consensus choice. We also discuss four important special cases of the general model and study several types of lower bounds for PRL reliability.Bruce Cooil is Associate Professor of Statistics, and Roland T. Rust is Professor and area head for Marketing, Owen Graduate School of Management, Vanderbilt University. The authors thank three anonymous reviewers and an Associate Editor for their helpful comments and suggestions. This work was supported in part by the Dean's Fund for Faculty Research of the Owen Graduate School of Management, Vanderbilt University.     

Evaluating interobserver reliability of interval data

Previous recommendations to employ occurrence, nonoccurrence, and overall estimates of interobserver reliability for interval data are reviewed. A rationale for comparing obtained reliability to reliability that would result from a random-chance model is explained. Formulae and graphic functions are presented to allow for the determination of chance agreement for each of the three indices, given any obtained per cent of intervals in which a response is recorded to occur. All indices are interpretable throughout the range of possible obtained values for the per cent of intervals in which a response is recorded. The level of chance agreement simply changes with changing values. Statistical procedures that could be used to determine whether obtained reliability is significantly superior to chance reliability are reviewed. These procedures are rejected because they yield significance levels that are partly a function of sample sizes and because there are no general rules to govern acceptable significance levels depending on the sizes of samples employed.     

Family process and child anxiety and aggression: An observational analysis

Barrett, Rapee, Dadds, and Ryan (1996) described a phenomenon whereby family discussions magnified the style of children's problem solving in a way characteristic of their particular clinical diagnosis. That is, anxious children became more avoidant, aggressive children more aggressive, and nonclinic children more prosocial, after discussing ambiguous hypothetical situations with their parents. This study examined specific sequences of communications exchanged between parents and children hypothesized to underlie this family exacerbation of child cognitive style. Family discussions were videotaped and categorized for groups of anxious, aggressive, and nonclinic children and their parents. Results revealed differences between groups of parents in frequency of agreeing with and listening to their child and the frequency of pointing out positive consequences. Conditional probability analyses showed that parents of anxious children were more likely to reciprocate avoidance, while parents of nonclinic children were more likely to agree with and listen to prosocial plans from their child. Differences in parent behaviors observed during the family discussions were reliably associated with the child's response to the ambiguous situation proposed after the family discussion. Results support a model of developmental anxiety and aggression that emphasizes the interaction of family processes and social-cognitive development in the child.This research was supported by grants from The National Health and Medical Research Council of Australia, The University of Queensland, and The Myer Foundation of Australia.     

The test-retest reliability of qualitative data

The test-retest reliability of qualitative items, such as occur in achievement tests, attitude questionnaires, public opinion surveys, and elsewhere, requires a different technique of analysis from that of quantitative variables. Definitions appropriate to the qualitative case are made both for the reliability coefficient of an individual on an item and for the reliability coefficient of a population on the item. From but a single trial of a large population on the item, it is possible to compute alower bound to the group reliability coefficient. Two kinds of lower bounds are presented. From two experimentally independent trials of the population on the item, it is possible to compute anupper bound to the group reliability coefficient. Two upper bounds are presented. The computations for the lower and upper bounds are all very simple. Numerical examples are given.     

ANSEC: A BASIC package for lag sequential analysis of observational data

Three BASIC programs for processing observational, nonconcurrent sequential data are presented. The programs follow Sackett’s lag sequential analysis method and have the innovations of running on interactional microcomputers and of providing plots of results. The outcome of the analysis is stored on a magnetic disk, facilitating a further application of probabilistic models to the transformed data. The Allison-Liker correction for the comparison test between expected and observed lag probabilities is included in the programs.     

A graphical method to aid the sequential analysis of observational data

In this paper, two methods of sequential analysis are applied to hypothetical observational data. The first method employs the conventional “conditional probability” approach, illustrated using the GSEQ program (Bakeman & Quera, 1995). In order to overcome some of the difficulties associated with the conditional probability approach, the second method employs a new “normalized and pooled” approach. Essentially, by normalizing periods of time preceding, during, and following each occurrence of a nominated “given” behavior, the proportion of time units devoted to a “target” behavior can be estimated and then pooled across all occurrences of the given behavior. A summary diagram representing the likelihood that the target behavior precedes, occurs concurrently with, and follows the given behavior can then be constructed. Elements of this summary diagram can also be quantified. Given the graphical nature of the output, and its ease of use, the normalized and pooled approach may help to promote the use of sequential analysis in applied settings.     

Dependence in the elderly from an interactional point of view: verbal and observational data

Dependence in the elderly was investigated from an interactional point of view. Observations as well as verbal data of the elderly and professional caregivers were considered. Samples consisted of 14 elderly people receiving regular home health care (M = 78.9, SD = 6.3 years) and 16 elderly people receiving regular nursing home care (M = 81.4, SD = 7.5 years). Results showed a strong tendency toward independence in the elderly. Sense of control in the self-care interactions observed was higher in the home health care group than in the group receiving nursing home care. Division of the elderly into subgroups with high, medium, and low perceived self-efficacy showed that the elderly high in self-efficacy were more independent in terms of observed self-care. Causal explanations of the elderly's competence and their nonuse of competence were significantly different between the elderly and staff.     

The style of processing scale: normative and reliability data.

Three student samples (ns=624, 653, and 566) yielded normative data for the Style of Processing scale. Analysis of subsamples found acceptable internal reliability (overall r=.78) and good test-retest reliability (overall r=.81, 6-mo. interval). The scale appears to have satisfactory reliability for assessing verbal-visual cognitive style, but further work is needed to test the scale's validity.