Quantifying false positives in simulated events using partial interval recording and momentary time sampling with dual‐criteria methods

The dual‐criteria (DC) and conservative dual‐criteria (CDC) methods allow clinicians and researchers to quantify the occurrence of false‐positive outcomes within single‐case experimental designs. The purpose of the current study was to use these DC and CDC methods to measure the incidence of false positives with simulated data collected via discontinuous interval methods (i.e., momentary time sampling, partial‐interval recording) as a function of data series length. We generated event data to create 316,800 unique simulated data series for analysis. In Experiment 1, we evaluated how changes in relevant parameters (i.e., interval sizes, event durations, IRT‐to‐event‐run ratios) produced false positives with momentary time sampling procedures. We also assessed the degree that the CDC method produced fewer false positives than the DC method with simulated interval data. In Experiment 2, we used similar procedures to quantify the occurrence of false positives with partial‐interval recording data. We successfully replicated outcomes from previous research in the current study, though such results only highlight the generality of the procedures relating to false positive (and not false negative) outcomes. That is, these results indicate MTS and PIR may adequately control for false positives, but our conclusions are limited by a lack of data on power.     

DETECTING CHANGES IN NON‐SIMULATED EVENTS USING PARTIAL INTERVAL RECORDING AND MOMENTARY TIME SAMPLING: EVALUATING FALSE POSITIVES,FALSE NEGATIVES,AND TRENDING

Several studies have evaluated false positives and false negatives produced with partial interval recording (PIR) and momentary time sampling (MTS) using simulated data. However, no study to date has evaluated false positives and negatives using a large sample of non‐simulated behaviors. In addition, few studies have evaluated whether interval methods of data collection alter trends that are evident in continuous records. We conducted three experiments to evaluate the extent to which various interval sizes of MTS and PIR produced false negatives (Experiment 1), false positives (Experiment 2), and trends that were inconsistent with the continuous records (Experiment 3). Collectively, the results show the following: (i) 10‐s PIR and 10‐s MTS produced few false negatives and few false positives (i.e., both were sensitive) to changes in duration events; (ii) 10‐s PIR produced very few false negatives, but an unexpected high percentage of false positives for frequency events; and (iii) each interval size of PIR and MTS produced a high percentage of changes in trending for duration events and frequency events. We briefly discuss the potential limitations and clinical implications of these findings. Copyright © 2012 John Wiley & Sons, Ltd.     

Using dual-criteria methods to supplement visual inspection: Replication and extension

The dual-criteria and conservative dual-criteria methods effectively supplement visual analysis with both simulated and published datasets. However, extant research evaluating the probability of observing false positive outcomes with published data may be affected by case selection bias and publication bias. Thus, the probability of obtaining false positive outcomes using these methods with data collected in the course of clinical care is unknown. We extracted baseline data from clinical datasets using a consecutive controlled case-series design and calculated the proportion of false positive outcomes for baseline phases of various lengths. Results replicated previous findings from Lanovaz, Huxley, and Dufour (2017), as the proportion of false positive outcomes generally decreased as the number of points in Phase B (but not Phase A) increased using both methods. Extending these findings, results also revealed differences in the rate of false positive outcomes across different types of baselines.     

Evaluating sources of baseline data using dual-criteria and conservative dual-criteria methods: A quantitative analysis

Scheithauer et al. (2020) recently demonstrated that differences in the source of baseline data extracted from a functional analysis (FA) may not affect subsequent clinical decision-making in comparison to a standard baseline. These outcomes warrant additional quantitative examination, as correspondence of visual analysis has sometimes been reported to be unreliable. In the current study, we quantified the occurrence of false positives within a dataset of FA and baseline data using the dual-criteria (DC) and conservative dual-criteria (CDC) methods. Results of this quantitative analysis suggest that false positives were more likely when using FA data (rather than original baseline data) as the initial treatment baseline. However, both sources of baseline data may have acceptably low levels of false positives for practical use. Overall, the findings provide preliminary quantitative support for the conclusion that determinations of effective treatment may be easily obtained using different sources of baseline data.     

Performance and process in collective and individual memory: the role of social decision schemes and memory bias in collective memory

To assess performance and processes in collective and individual memory, participants watched two job candidates on video. Beforehand, half the participants were told they would be tested on their memory of the interviews, and the other half were asked to make a decision to hire one of the candidates. Afterwards, participants completed a recognition memory task in either a group or individual condition. Groups had better recognition memory than individuals. Individuals made more false positives than false negatives and groups exaggerated this. Post-hoc analysis found that groups only exaggerated the tendency towards false positives on items that reflected negatively on the job candidate. There was no significant difference between instruction conditions. When reaching consensus on the recognition task, groups tended to choose the correct answer if at least two members had the correct answer. This method of consensus is discussed as a factor in groups' superior memory performance.     

Implicit Learning of an Invariant: Just Say No

Following exposure to 30 four-digit numbers containing an invariant “3”, subjects are found falsely to recognize novel four-digit numbers containing this invariant (positives) in preference to novel numbers that do not contain the invariant (negatives). Despite this false recognition, they are generally unable to report the rule relating test positives to the positives seen during the learning phase. This finding has been taken to show implicit learning of a rule. Two experiments are reported here which show that it is not necessary to learn this rule in order to perform at above-chance levels on this test. Most of the effect can be explained in terms of the rejection of particularly distinctive test items that are more prevalent in the test negatives. This rejection appears to be mediated by knowledge that is potentially explicit as opposed to implicit, and we present tentative evidence that it is rule-based as opposed to analogic.     

A semi-parametric within-subject mixture approach to the analyses of responses and response times

In item response theory, modelling the item response times in addition to the item responses may improve the detection of possible between- and within-subject differences in the process that resulted in the responses. For instance, if respondents rely on rapid guessing on some items but not on all, the joint distribution of the responses and response times will be a multivariate within-subject mixture distribution. Suitable parametric methods to detect these within-subject differences have been proposed. In these approaches, a distribution needs to be assumed for the within-class response times. In this paper, it is demonstrated that these parametric within-subject approaches may produce false positives and biased parameter estimates if the assumption concerning the response time distribution is violated. A semi-parametric approach is proposed which resorts to categorized response times. This approach is shown to hardly produce false positives and parameter bias. In addition, the semi-parametric approach results in approximately the same power as the parametric approach.     

A Review of the Office of Technology Assessment Report on Polygraph Validity

This article reviews the assessment of polygraphic lie detection produced by the Office of Technology Assessment (OTA) in November 1983. The review argues that use of the lambda statistic to summarize polygraph accuracy in the OTA report was inappropriate because the studies examined differ widely in base rates of guilt and innocence. Using Lykken's (1981) average accuracy statistic and avoiding overlapping data sets found in the OTA report, the review finds that field studies, analog studies, and guilty knowledge studies produce very similar average accuracy (82% to 88%). Guilty knowledge tests tend to be biased toward false negatives, whereas control question tests tend to be biased toward false positives. The striking similarity of results for field and analog studies using control question tests contradicts the common belief that results of analog testing are weak and not generalizable to field practice. It is argued that analog studies are in fact the only hope for improving the validity of polygraph testing.     

The Role of Incentive Framing on Training and Transfer of Learning in a Visual Threat Detection Task

We examined the effects of different incentives on skill acquisition and transfer during threat detection in airline luggage screening. The incentives were presented within positive (gains) or negative (losses) frames, and points were given or taken away accordingly during training (with familiar targets) and transfer (to novel targets). During training, incentives exerted a more beneficial effect on skill acquisition than training without incentives. During transfer, incentives benefitted performance largely when presented as losses or penalties. Incentives framed as gains primed participants to say ‘yes’ more often leading to a high ratio of false positives; however, incentives framed as losses lead participants to become more selective in their ‘yes’ responses leading to a lower number of false positives but a comparable probability of correct detections. Interestingly, participants that received no training outperformed participants that received incentive‐based training, suggesting that incentives actually constrained rather than helped transfer of learning in this study. Copyright © 2011 John Wiley & Sons, Ltd.     

A closer look at self-reported suicide attempts: false positives and false negatives

The validity of self-reported suicide attempt information is undermined by false positives (e.g., incidences without intent to die), or by unreported suicide attempts, referred to as false negatives. In a sample of 1,385 Austrian adults, we explored the occurrence of false positives and false negatives with detailed, probing questions. Removing false positives decreased the rate of suicide attempters from 4.3% to 2.7%. Probing questions also revealed 0.8% false negatives. We recommend using probing questions with both those who report a suicide attempt and those who do not report a suicide attempt to increase the validity of self-reported suicide-related information.     

Alternative-based thresholding with application to presurgical fMRI

Functional magnetic reasonance imaging (fMRI) plays an important role in pre-surgical planning for patients with resectable brain lesions such as tumors. With appropriately designed tasks, the results of fMRI studies can guide resection, thereby preserving vital brain tissue. The mass univariate approach to fMRI data analysis consists of performing a statistical test in each voxel, which is used to classify voxels as either active or inactive—that is, related, or not, to the task of interest. In cognitive neuroscience, the focus is on controlling the rate of false positives while accounting for the severe multiple testing problem of searching the brain for activations. However, stringent control of false positives is accompanied by a risk of false negatives, which can be detrimental, particularly in clinical settings where false negatives may lead to surgical resection of vital brain tissue. Consequently, for clinical applications, we argue for a testing procedure with a stronger focus on preventing false negatives. We present a thresholding procedure that incorporates information on false positives and false negatives. We combine two measures of significance for each voxel: a classical p-value, which reflects evidence against the null hypothesis of no activation, and an alternative p-value, which reflects evidence against activation of a prespecified size. This results in a layered statistical map for the brain. One layer marks voxels exhibiting strong evidence against the traditional null hypothesis, while a second layer marks voxels where activation cannot be confidently excluded. The third layer marks voxels where the presence of activation can be rejected.     

Indirect Effects of Functional Communication Training on Non-Targeted Disruptive Behavior

The purpose of this study was to evaluate the effects of functional communication training (FCT) on the occurrence of non-targeted disruptive behavior. The 10 participants were preschool-aged children with developmental disabilities who engaged in both destructive (property destruction, aggression, self-injury) and disruptive (hand flapping, spinning in circles, shrill laughter, screaming, crying) behaviors. Only destructive behavior was targeted for the functional analyses and FCT, but data were also collected on disruptive behaviors. All procedures were conducted in the participants’ homes by their mothers with investigator coaching. Phase 1 consisted of conducting a functional analysis within a multielement design. Phase 2 consisted of conducting FCT with demand fading and repeated extinction baselines within a reversal design. Single-case data are provided for 3 participants, and summary data are provided for all 10 participants. Results of phase 1 showed that all participants’ destructive and disruptive behavior was maintained, at least in part, by negative reinforcement. Results of phase 2 showed that both destructive behavior and non-targeted disruptive behavior occurred at lower levels during FCT when compared to the functional analysis demand condition and baseline conditions, suggesting that FCT was effective in decreasing both target destructive behavior and non-targeted disruptive behaviors.     

The novelty effect: support for the Novelty-Encoding Hypothesis

In two experiments, we examined the "Novelty-Encoding Hypothesis" proposed by Tulving and Kroll (1995), suggesting that the encoding of online information into long-term memory is influenced by its novelty and that novelty increases recognition performance. In Phase 1 (familiarization phase), subjects participated in a standard memory experiment in which different types of materials (verbs and nouns) were studied under different encoding conditions (enactment and non-enactment) and were tested by an expected recognition test. In Phase 2 (critical phase), subjects evaluated the materials (both familiar materials which were encoded earlier in Phase 1, and novel materials which were not presented earlier in Phase 1) in a frequency judgment task and were given an unexpected recognition test. The results of both experiments showed that novel items were recognized better than familiar items. This result held true for both hit rates - false alarms and hit rates. The novelty effect was observed for different subjects (Swedish and Japanese), different materials (verbs and nouns; high frequency and low frequency), and different types of encoding in Phase 1 (enactment and non-enactment). These findings provide support for the "Novelty-Encoding Hypothesis" stating that the effect is based on the encoding of target items at the time of the critical study (Phase 2). A comparison between the present experiments and the Tulving and Kroll (1995), Dobbins, Kroll, Yonelinas & Liu (1998) and Greene (1999) studies suggests that the novelty effect is more pronounced under incidental encoding than under intentional encoding.     

Should I test more babies? Solutions for transparent data peeking

Research with infants is often slow and time-consuming, so infant researchers face great pressure to use the available participants in an efficient way. One strategy that researchers sometimes use to optimize efficiency is data peeking (or “optional stopping”), that is, doing a preliminary analysis (whether a formal significance test or informal eyeballing) of collected data. Data peeking helps researchers decide whether to abandon or tweak a study, decide that a sample is complete, or decide to continue adding data points. Unfortunately, data peeking can have negative consequences such as increased rates of false positives (wrongly concluding that an effect is present when it is not). We argue that, with simple corrections, the benefits of data peeking can be harnessed to use participants more efficiently. We review two corrections that can be transparently reported: one can be applied at the beginning of a study to lay out a plan for data peeking, and a second can be applied after data collection has already started. These corrections are easy to implement in the current framework of infancy research. The use of these corrections, together with transparent reporting, can increase the replicability of infant research.     

The relation between lisping and visual recognition memory in children

It was argued that if silent reading involves articulatory coding then children with a pronounced lisp ought to code two visual nonsense words such as RETH and RESS more similarly than children without a lisp. This, in tum, would lead to more false positives in a recognition memory task. Although the data confirmed this hypothesis, it turned out that lisping children made more correct positives (hits) as well. Lisping and normal children did not differ in sensitivity (d^′); therefore, the main result was a difference in criterion. An implausible hypothesis was offered so as to salvage a portion of the original argument.     

The long and short of it: Reports of pause occurrence and duration in speech

Native speakers of American English and of German listened to a passage several minutes in duration in each language and reported occurrence and duration of pauses. Subjects overestimated occurrence (false positives) and understimated duration. Overall analyses of variance indicated that Germans reported a higher percentage of pauses than Americans; that a higher percentage of pauses was reported in native than in nonnative passages; that Americans reported a higher percentage of pauses in English passages, whereas Germans' percentages were the same in both languages; that American women reported more false positives than American men, whereas German men reported more than German women; and that Americans and Germans both reported more false positives in English than in German. Detailed analyses of the individual passages yielded reliable differences between Germans and Americans in their reports of pauses in three duration brackets: .3;,<1.0;1.0 sec. Pause reports of this kind diverge from objectively measured pause data as a function of a number of independent variables and are therefore not to be trusted as objective estimates of either pause occurrence or pause duration.     

Risk of false positives when identifying malingered profiles using the trauma symptom inventory

The Trauma Symptom Inventory (TSI; Briere, 1995) is a 100-item self-report measure of posttraumatic symptomatology that includes an Atypical Response (ATR) validity scale designed to differentiate honest from malingered profiles. In this study, using an analogue design to experimentally manipulate honest and malingered responses on the TSI, we found that proposed ATR cut scores produce a significant risk of false positives. Furthermore, the functioning of proposed cut scores worsened when we used posttraumatic stress disorder relevant samples and low estimates of malingering base rates. In light of these findings, the TSI should be used with caution when assessing claims of posttraumatic stress in forensic or disability settings.     

Effects of standard modality, comparison modality and retention interval on matching of form

Garvill, J. & Molander, B. Effects of standard modality, comparison modality and retention interval on matching of form. Scand. J. Psychol., 1973, 14, 203–206.MdashIntra-modal and cross-modal matching of nonsense forms was studied in a 2 (standard modalities: visual vs. tactual) by 2 (comparison modalities: visual vs. tactual) by 3 (intervals between standard and comparison: 1, 10 and 30 sec) factorial experiment. The errors were divided into false negatives and false positives. Significant effects of standard modality and of comparison modality were found for false negatives. For false positives the most prominent effect was an interaction between the standard modality and the comparison modality. Retention interval had no effect in any of the modality conditions. The effects are discussed in terms of differential information processing capacity for the visual and the tactual modalities.     

情绪刺激在虚假记忆产生中的作用:来自ERP与眼动的联合证据

摘 要 为探索情绪刺激如何影响虚假记忆及其在分离虚假记忆与真实记忆神经活动中的作用,以IAPS图片为标准情绪刺激,19名大学生为被试,使用DRM实验范式,从行为、眼动和ERP三个方面进行了联合分析。结果发现,情绪的唤醒度和效价都能促使虚假记忆的产生,只是唤醒度影响时程更早。眼动和ERP的联合分析还发现了在情绪刺激下虚假记忆与真实记忆在神经活动层面上存在差异：早期虚假再认图片比真实再认图片需要更深的加工来实现对目标的识别;晚期虚假再认需要更多的信息搜索、比较等加工来完成其决策行为。     

A nonparametric procedure for evaluation of true and false positives

A procedure is presented for the evaluation of single-point estimates of true and false positives without strong underlying parametric assumptions. The method is based upon the area operating characteristic and the Green area rule. Estimates of sampling error are also available. The procedure is extended to a strong one-parameter relation between true and false positives.