An ecological study of the reliability of clinical judgment in executive appraisal

This research was a study of the reliability of clinical judgment findings (multitrait) across three different information sources (psychometric tests, structured interview, and psychometric tests and interview). Subjects (N = 74) were middle and senior executives of Western Canadian technical companies; clinicians (N = 3) were trained and experienced industrial psychologists. The study investigated the similarity of clinical evaluation of personological characteristics (based on an 18-factor multitrait paradigm) across the three different information sources. Subjects were independently rated by a single clinician on 18 criterion factors in each of the three information source categories. Test information source categories required the administration of approximately 12 hrs of standardized psychological assessment questionnaires to each of the 74 subjects. Interview source category involved a 1.5-hr structured interview per subject. Combined condition pooled both test and interview conditions. Generalizability of the findings was maximized by the undertaking of the experiment in a natural situation thus increasing ecological validity. Statistical treatments used were designed to assess the similarity of a clinician's evaluation of a subject based on the different category of information available about that client. Convergence (intrarater reliability) indexes range from a high of .64 to a low of .05. Results indicate a varying degree of convergence of multitrait clinical ratings dependent on clinician and trait being rated. Results are discussed in terms of implications for practitioners involved in executive personnel selection.     

THE BEHAVIOR OBSERVATION INSTRUMENT: A METHOD OF DIRECT OBSERVATION FOR PROGRAM EVALUATION1

The background and development of a multicategory direct observation system, the Behavior Observation Instrument (BOI), is described. This time-sampling procedure for recording the behavior of persons is demonstrated in several treatment settings and the results applied to issues of program evaluation. Elements that have prevented direct observation from being widely adopted, such as costs, manpower, and training requirements, are systematically analyzed. A basic psychometric analysis of the instrument is used to determine optimum frequency and duration of observation intervals as well as observer agreement. The results imply that direct observation methods, once assumed by some to belong to the special province of the single-subject design, can be used to assess the effects of programs on groups of psychiatric clients in an efficient and economic manner.     

Behavioral self-control in classroom settings: A review of the literature

Reviews the research on the use of behavioral self-control procedures with students in classroom settings. Thirteen published articles are reviewed. The three student classroom behaviors of on-task behavior, disruptive behavior, and academic behavior are considered. The use of behavioral self-control procedures appears promising with on-task and academic behaviors and inconclusive with disruptive behavior. The reviewed studies were weak regarding follow-up analyses and controlling for student prior experience with external contigency programs. Further research with behavioral self-control in different settings seems warranted.     

Validity of psychologists' identification of EMR students in the perspective of the California decertification experience

The Diana (Diana v. State Board of Education, Note 1) and the Larry P. (Larry P. v. Riles, Note 2) litigations led to decertification of thousands of California EMR students and their return to the regular program. Allegations, including those of biased testing, went largely uncontested in order that change in special education could result from court mandate, but the allegations left the psychologists in a libeled state. A study of cumulative records and psychologists files in 12 representative districts permitted a comparison of the EMR placement of those later decertified (D) with matched nondecertified (EMR). Other than a small mean difference in IQ at placement, nothing was found to support various allegations. Records showed no systematic differences betweeen D and EMR groups in pre-EMR reasons for referral nor in teacher marks, discnfirming beliefs in referral of higher functioning students for deportment rather than for academic failure. Districts permitted a mean of at least two years in regular placement before assessment, attesting to class failure rather than IQ as the initial and necessary basis for EMR identification (no D-EMR difference in this either). The data permit a conclusion that the work of the school psychologists in the EMR placement was professionally competent, given the guidelines in effect at the time.     

Relative costs and statistical power in the extreme groups approach

The relationship between variables in applied and experimental research is often investigated by the use of extreme (i.e., upper and lower) groups. Recent analytical work has provided an extreme groups procedure that is more powerful than the standard correlational approach for all values of the correlation and extreme group size. The present article provides procedures to optimize power by determining the relative number of subjects to use in each of two stages of data collection given a fixed testing budget.The opinions or assertions contained herein are those of the writers and are not to be construed as official or reflecting the views of the Navy Department.Edward F. Alf, Jr. is also affiliated with the Department of Psychology, San Diego State University.