ActSds and OdfSds: Programs for converting INTERACT and The Observer data files into SDIS timed-event sequential data files

In this article, we describe programs for converting Mangold International's INTERACT and Noldus Information Technology's The Observer data files to sequential data interchange standard (SDIS) timed-event sequential data files. Users who convert their INTERACT or The Observer data files can then take advantage of various flexible and powerful data modification and computational procedures available in the Generalized Sequential Querier, a program that assumes SDIS-formatted files.     

Observer agreement, credibility, and judgment: some considerations in presenting observer agreement data

Graphical and statistical indices employed to represent observer agreement in interval recording are described as "judgmental aids", stimuli to which the researcher and scientific community must respond when viewing observer agreement data. The advantages and limitations of plotting calibrating observer agreement data and reporting conventional statistical aids are discussed in the context of their utility for researchers and research consumers of applied behavior analysis. It is argued that plotting calibrating observer data is a useful supplement to statistical aids for researchers but is of only limited utility for research consumers. Alternatives to conventional per cent agreement statistics for research consumers include reporting special agreement estimates (e.g., per cent occurrence agreement and nonoccurrence agreement) and correlational statistics (e.g., Kappa and Phi).     

Comparing time-based and event-based prospective memory over short delays

The current study compared monitoring in time- and event-based prospective memory (PM). Time- and event-based non-focal task instructions were given after a baseline block of a lexical decision ongoing task. Delay between instruction and presentation of PM cue/time was manipulated between-subjects to examine monitoring across short delays (1–6?min). Longer delays decreased performance in the event-based task, but not in the time-based task. This accuracy decline was accompanied by a decline in monitoring (as measured by PM cost to the ongoing task in the trials immediately before the PM cue was presented) between the 1 and 3?min delays. Monitoring was only evident for the time-based task at the 6?min delay as measured by PM cost to the ongoing task. Clock checks were also not affected by delay, but did increase in frequency as the response time neared. These results suggest that delay from the time of intention formation decreases both accuracy and monitoring in event-based tasks, but does not decrease accuracy or monitoring in time-based tasks.     

The age prospective memory paradox within the same sample in time-based and event-based tasks

The present research investigated the age prospective memory (PM) paradox by testing the performance of the same participants on laboratory and naturalistic PM tasks. Younger, middle-aged, and older adults performed three tasks (time-based, event-based with focal cue, and event-based with nonfocal cue); first in the laboratory, then in the context of their everyday lives. Additionally, the social importance of PM tasks was manipulated in the laboratory. As expected, age-dependent declines on the laboratory tasks were reversed in the naturalistic tasks. Middle-aged adults performed as well as younger adults in the laboratory and as well as the elderly outside of the laboratory. When the social importance of laboratory tasks was stressed, the performance of younger adults fell. In addition, older adults showed higher self-reported commitment to the naturalistic tasks than both younger and middle-aged adults. Findings are discussed in the context of possible explanations for the age PM paradox.     

Observer agreement for event sequences: Methods and software for sequence alignment and reliability estimates

When sequences of discrete events, or other units, are independently coded by two coders using a set of mutually exclusive and exhaustive codes, but the onset times for the codes are not preserved, it is often unclear how pairs of protocols should be aligned. Yet such alignment is required before Cohen’s kappa, a common agreement statistic, can be computed. Here we describe a method—based on the Needleman and Wunsch (1970) algorithm originally devised for aligning nucleotide sequences—for optimally aligning such sequences; we also offer the results of a simulation study of the behavior of alignment kappa with a number of variables, including number of codes, varying degrees of observer accuracy, sequence length, code variability, and parameters governing the alignment algorithm. We conclude that (1) under most reasonable circumstances, observer accuracies of 90% or better result in alignment kappas of .60 or better; (2) generally, alignment kappas are not strongly affected by sequence length, the number of codes, or the variability in the codes’ probability; (3) alignment kappas are adversely affected when missed events and false alarms are possible; and (4) cost matrices and priority orders used in the algorithm should favor substitutions (i.e., disagreements) over insertions and deletions (i.e., missed events and false alarms). Two computer programs were developed: Global Sequence Alignment, or GSA, for carrying out the simulation study, and Event Alignment, or ELign, a user-oriented program that computes alignment kappa and provides the optimal alignment given a pair of event sequences.     

The Observer: A software system for collection and analysis of observational data

The Observer is a general-purpose software package for event recording and data analysis in behavioral research. It allows any IBM-type personal computer to serve as an event recorder. In addition, The Observer can generate dedicated event-recording programs for several types of non-IBM-compatible portable and hand-held computers and transfer files between the PC and such computers. The user specifies options through menus. The configuration can be either used directly for event recording on the PC or passed on to a program generator that creates a program to collect data on a hand-held computer. Observational data from either type of computer can be analyzed by the program. Event-recording configurations can be tailored to many different experimental designs. Keys can be designated as events, and modifiers can be used to indicate the limits of an event. The program allows grouping of events in classes and distinction between mutually exclusive versus nonexclusive events and duration events versus frequency events. Timing of events is accurate to 0.1 sec. An on-line electronic notepad permits notes to be made during an observation session. The program also includes on-line error correction. User comments as well as independent variables can be stored together with the observational data. During data analysis, the user can select the level of analysis and the type of output file. The Observer calculates frequency of occurrence and duration for classes of events, individual events, or combinations of events. For analysis of concurrence, one can select the number of nesting levels and the order of nesting. Output can be generated in the form of sorted event sequence files, text report files, and tabular ASCII files. The results can be exported to spreadsheet and statistical programs.     

C-quence: A tool for analyzing qualitative sequential data

C-quence is a software application that matches sequential patterns of qualitative data specified by the user and calculates the rate of occurrence of these patterns in a data set. Although it was designed to facilitate analyses of face-to-face interaction, it is applicable to any data set involving categorical data and sequential information. C-quence queries are constructed using a graphical user interface. The program does not limit the complexity of the sequential patterns specified by the user.     

Interrater reliability and agreement of performance ratings: A methodological comparison

This paper demonstrates and compares methods for estimating the interrater reliability and interrater agreement of performance ratings. These methods can be used by applied researchers to investigate the quality of ratings gathered, for example, as criteria for a validity study, or as performance measures for selection or promotional purposes. While estimates of interrater reliability are frequently used for these purposes, indices of interrater agreement appear to be rarely reported for performance ratings. A recommended index of interrater agreement, theT index (Tinsley & Weiss, 1975), is compared to four methods of estimating interrater reliability (Pearsonr, coefficient alpha, mean correlation between raters, and intraclass correlation). Subordinate and superior ratings of the performance of 100 managers were used in these analyses. The results indicated that, in general, interrater agreement and reliability among subordinates were fairly high. Interrater agreement between subordinates and superiors was moderately high; however, interrater reliability between these two rating sources was very low. The results demonstrate that interrater agreement and reliability are distinct indices and that both should be reported. Reasons are discussed as to why interrater reliability should not be reported alone.This paper is based, in part, on a thesis submitted to East Carolina University by the second author. Portions of this study were presented at the American Psychological Association meeting in New Orleans, LA, August, 1989. The authors would like to thank Michael Campion and two anonymous reviewers for their comments on earlier drafts of this paper.     

SDIS: A sequential data interchange standard

Few general-purpose computer programs are available that analyze sequential categorical data. If there were a sequential data interchange standard—a standard way of representing sequential data—then it would be more attractive to write general-purpose computer programs for such data Moreover, interlaboratory sharing would be facilitated. The present paper defines such a standard, called the sequential data interchange standard, or SDIS. Both the SDIS data language and a parsing program for data that follow SDIS conventions are described. The parsing program will be made available to researchers who wish to develop analysis programs for sequential data     

OTS: A program for converting Noldus Observer data files to SDIS files

A program for converting Noldus Observer data files (ODF) to sequential data interchange standard (SDIS) files is described. Observer users who convert their data files can then take advantage of various flexible and powerful data modification and computational procedures available in the Generalized Sequential Querier, a program that assumes SDIS-formatted files.     

A note on interobserver reliability for sequential data

It is well accepted that reliability measures based on simple frequency counts of designated codes are inappropriate for sequential analysis. The stricter point-by-point method, however, is problematic, particularly because it is too strict. It is suggested that a less strict reliability measure based on the number of transitions be used. This method also avoids problems encountered with the point-by-point method and another less stringent method.     

Systema Temporis: A time-based dimensional framework for consciousness and cognition

This study uses a combined categorical-dimensional approach to depict a hierarchical framework for consciousness similar to, and contiguous with, factorial models of cognition (cf., intelligence). On the basis of the longstanding definition of time consciousness, the analysis employs a dimension of temporal extension, in the same manner that psychology has temporally organised memory (i.e., short-term, long-term, and long-lasting memories). By defining temporal extension in terms of the structure of time perception at short timescales (<100 s), memory and time consciousness are proposed to fit along the same logarithmic dimension. This suggests that different forms of time consciousness (e.g., experience, wakefulness, and self-consciousness) are embedded within, or supported by, the ascending timescales of different modes of memory (i.e., short-term, long-term, etc.). A secondary dimension is also proposed to integrate higher-order forms of consciousness/emotion and memory/cognition. The resulting two-dimensional structure accords with existing theories of cognitive and emotional intelligence.     

ANSEC: A BASIC package for lag sequential analysis of observational data

Three BASIC programs for processing observational, nonconcurrent sequential data are presented. The programs follow Sackett’s lag sequential analysis method and have the innovations of running on interactional microcomputers and of providing plots of results. The outcome of the analysis is stored on a magnetic disk, facilitating a further application of probabilistic models to the transformed data. The Allison-Liker correction for the comparison test between expected and observed lag probabilities is included in the programs.     

Monotonic measures of agreement for ranked data

The τ_b and y statistics are interpreted as rank-monotonic coefficients of partial agreement. Using a method of transposition employed by Pearson's r_i intraclass correlation coefficient, the τ_bi and y_i intraclass coefficients of total monotonic agreement are created. Transpositional measures of agreement like τ_bi and τ_i measure the combined effects of cell and marginal disagreement which make them particularly suitable for reliability studies. The coefficients are also made applicable to K > 2 sets of ranks.     

A comparison of sequential sampling models for two-choice reaction time

The authors evaluated 4 sequential sampling models for 2-choice decisions--the Wiener diffusion, Ornstein-Uhlenbeck (OU) diffusion, accumulator, and Poisson counter models--by fitting them to the response time (RT) distributions and accuracy data from 3 experiments. Each of the models was augmented with assumptions of variability across trials in the rate of accumulation of evidence from stimuli, the values of response criteria, and the value of base RT across trials. Although there was substantial model mimicry, empirical conditions were identified under which the models make discriminably different predictions. The best accounts of the data were provided by the Wiener diffusion model, the OU model with small-to-moderate decay, and the accumulator model with long-tailed (exponential) distributions of criteria, although the last was unable to produce error RTs shorter than correct RTs. The relationship between these models and 3 recent, neurally inspired models was also examined.     

Analyzing sequential relations in observational data: A practical guide

Many observational systems in basic and applied research produce a record of sequences of events over time. Within such observational systems, important information may be found in the frequency of the transitions between events that does not emerge in the typical researcher's focus on absolute event frequency. Indeed, if many behaviors are controlled by closely adjacent preceding events, then substantial prediction and control can be obtained through knowledge and manipulation of causal event transitions. A method for analyzing and testing sequential dependencies between events is proposed as part of an integrated package of computer-based data entry, storage, and analysis procedures. The mathematical portion of these techniques is based on the statistic, kappa,which is applicable to determining whether particular transitions among events differ from chance and whether particular transitions differ significantly across groups of subjects. Low-cost hardware and software to implement the proposed procedures are described.This work has been supported by Grant 1 RO1 HD19245-01A1 from the National Institute of Child Health and Human Development to the two senior authors and by grants of equipment from Commodore Computers, Inc., Koala Corporation, and NEC Telephones, Inc.     

Dismal: a spreadsheet for sequential data analysis and HCI experimentation

Dismal is a spreadsheet that works within GNU Emacs, a widely available programmable editor. Dismal has three features of particular interest to those who study behavior: (1) the ability to manipulate and align sequential data, (2) an open architecture that allows users to expand it to meet their particular needs, and (3) an instrumented and accessible interface for studies of human-computer interaction (HCI). Example uses of each of these capabilities are provided, including cognitive models that have had their sequential behavior aligned with subject’s protocols, extensions useful for teaching and doing HCI design, and studies in which keystroke logs from the timing package in Dismal have been used.     

A class of k-modes algorithms for extracting knowledge structures from data

One of the most crucial issues in knowledge space theory is the construction of the so-called knowledge structures. In the present paper, a new data-driven procedure for large data sets is described, which overcomes some of the drawbacks of the already existing methods. The procedure, called k-states, is an incremental extension of the k-modes algorithm, which generates a sequence of locally optimal knowledge structures of increasing size, among which a “best” model is selected. The performance of k-states is compared to other two procedures in both a simulation study and an empirical application. In the former, k-states displays a better accuracy in reconstructing knowledge structures; in the latter, the structure extracted by k-states obtained a better fit.     

Judgments of moderator variables of agreement in person perception: A comparison of two cultures

Three moderators of agreement in person perception, behavioral consistency, observability and social desirability, were studied. The major hypothesis is that the moderators can be estimated using the standing of targets on traits; that is, that as targets vary on a given trait, they vary on how they are seen as on the moderators. Using Korean ( N  = 135) and US ( N  = 81) samples, we tested this approach for 80 traits. Analyses revealed that moderators varied by the combination of trait and target standing in different ways for the two samples. In judgment of behavioral consistency over target standings, linear and curvilinear trends were stronger for the US sample than for the Korean sample. For observability, judgments were similar, although curvilinear trends were larger for the Korean sample. Furthermore, being extremely positive was perceived as less desirable for the Korean judges. These findings were discussed in terms of cultural differences. Moreover, a new approach to the study of moderators was proposed.