Asymptotic log-linear analysis: some cautions concerning sparse frequency tables

Traditional asymptotic probability values resulting from log-linear analyses of sparse frequency tables are often much too large. Asymptotic probability values for chi-squared and likelihood-ratio statistics are compared to nonasymptotic and exact probability values for selected log-linear models. The asymptotic probability values are all too often substantially larger than the exact probability values for the analysis of sparse frequency tables. An exact nondirectional permutation method is presented to analyze combined independent multinomial distributions. Exact nondirectional permutation methods to analyze hypergeometric distributions associated with r-way frequency tables are confined to r = 2.     

Exact and resampling probability values for measures associated with ordered r by c contingency tables

Permutation procedures to compute exact and resampling probability values associated with measures of association for ordered r x c contingency tables are described. Comparisons with asymptotic probability values demonstrated that exact and resampling permutation procedures were preferred for sparse contingency tables.     

Multiway contingency tables: Monte Carlo resampling probability values for the chi-squared and likelihood-ratio tests

Monte Carlo resampling methods to obtain probability values for chi-squared and likelihood-ratio test statistics for multiway contingency tables are presented. A resampling algorithm provides random arrangements of cell frequencies in a multiway contingency table, given fixed marginal frequency totals. Probability values are obtained from the proportion of resampled test statistic values equal to or greater than the observed test statistic value.     

Resampling programs for multiway contingency tables with fixed marginal frequency totals

A resampling algorithm is presented for analyzing multiway contingency tables with fixed marginal frequency totals. Applications are illustrated with extensions of Fisher's exact, Pearson's chi-squared, and likelihood-ratio tests to three-way contingency tables.     

Exact and resampling probability values for weighted kappa

Permutation procedures to compute exact and resampling probability values for weighted kappa are described. Comparisons with asymptotic probability values demonstrate that exact permutation procedures are advantageous for sparse data sets, whereas resampling permutation procedures are appropriate for both sparse and nonsparse data sets.     

A measure of effect size for r x c contingency tables

Goodman and Kruskal's tau measure of categorical association is advanced as a replacement for conventional measures of effect size for r x c contingency tables. Goodman and Kruskal's tau is an asymmetric measure of categorical association which is based entirely on the observed data and possesses a clear interpretation in terms of proportional reduction in error. Comparisons with conventional measures of effect size based on chi-squared such as Pearson's phi2, Tschuprow's T2, and Cramer's V2 demonstrate the advantages of employing tau as a measure of effect size.     

A note on the use of mood's likelihood ratio test for item analyses involving 2 × 2 tables with small samples

Mood's likelihood ratio test is generally considered an unreliablex ² approximation in 2 × 2 contingency tables containing expected cell frequencies less than five. Probability values were computed for 60 such tables as part of an item analysis for two 30-item alternate forms of a measure. The rank orders of the items, from best to worst differentiators, as determined separately by Mood's test and by Fisher's exact test correlated .97 for one form and .96 for the other.     

Exact procedures for the analysis of multidimensional contingency tables

The log-linear model for contingency tables expresses the logarithm of a cell frequency as an additive function of main effects, interactions, etc., in a way formally identical with an analysis of variance model. Exact statistical tests are developed to test hypotheses that specific effects or sets of effects are zero, yielding procedures for exploring relationships among qualitative variables which are suitable for small samples. The tests are analogous to Fisher's exact test for a 2 × 2 contingency table. Given a hypothesis, the exact probability of the obtained table is determined, conditional on fixed marginals or other functions of the cell frequencies. The sum of the probabilities of the obtained table and of all less probable ones is the exact probability to be considered in testing the null hypothesis. Procedures for obtaining exact probabilities are explained in detail, with examples given.     

Exact probabilities for contingency tables using binomial coefficients

The use of binomial coefficients in place of factorials to shorten the calculation of exact probabilities for 2 × 2 and 2 ×r contingency tables is discussed. A useful set of inequalities for estimating the cumulative probabilities in the tail of the distribution from the probability of a single table is given. A table of binomial coefficients with four significant places andn through 60 is provided.     

Combining probability values from independent permutation tests: a discrete analog of Fisher's classical method

Permutation tests are based on all possible arrangements of observed data sets. Consequently, such tests yield exact probability values obtained from discrete probability distributions. An exact nondirectional method to combine independent probability values that obey discrete probability distributions is introduced. The exact method is the discrete analog to Fisher's classical method for combining probability values from independent continuous probability distributions. If the combination of probability values includes even one probability value that obeys a sparse discrete probability distribution, then Fisher's classical method may be grossly inadequate.     

Limited‐information goodness‐of‐fit testing of item response theory models for sparse 2P tables

Bartholomew and Leung proposed a limited‐information goodness‐of‐fit test statistic (Y) for models fitted to sparse 2^P contingency tables. The null distribution of Y was approximated using a chi‐squared distribution by matching moments. The moments were derived under the assumption that the model parameters were known in advance and it was conjectured that the approximation would also be appropriate when the parameters were to be estimated. Using maximum likelihood estimation of the two‐parameter logistic item response theory model, we show that the effect of parameter estimation on the distribution of Y is too large to be ignored. Consequently, we derive the asymptotic moments of Y for maximum likelihood estimation. We show using a simulation study that when the null distribution of Y is approximated using moments that take into account the effect of estimation, Y becomes a very useful statistic to assess the overall goodness of fit of models fitted to sparse 2^P tables.     

A test for psychometric function shift

A nonparametric, small-sample-size test for the homogeneity of two psychometric functions against the left- and right-shift alternatives has been developed. The test is designed to determine whether it is safe to amalgamate psychometric functions obtained in different experimental sessions. The sum of the lower and upper p-values of the exact (conditional) Fisher test for several 2 × 2 contingency tables (one for each point of the psychometric function) is employed as the test statistic. The probability distribution of the statistic under the null (homogeneity) hypothesis is evaluated to obtain corresponding p-values. Power functions of the test have been computed by randomly generating samples from Weibull psychometric functions. The test is free of any assumptions about the shape of the psychometric function; it requires only that all observations are statistically independent.     

Canonical correlation and chi-square: relationships and interpretation

A 2 x 2 chi-square can be computed from a phi coefficient, which is the Pearson correlation between two binomial variables. Similarly, chi-square for larger contingency tables can be computed from canonical correlation coefficients. The authors address the following series of issues involving this relationship: (a) how to represent a contingency table in terms of a correlation matrix involving r - 1 row and c - 1 column dummy predictors; (b) how to compute chi-square from canonical correlations solved from this matrix; (c) how to compute loadings for the omitted row and column variables; and (d) the possible interpretive advantage of describing canonical relationships that comprise chi-square, together with some examples. The proposed procedures integrate chi-square analysis of contingency tables with general correlational theory and serve as an introduction to some recent methods of analysis more widely known by sociologists.     

Exact tests,confidence regions and estimates

This paper proposes a uniform method for constructing tests, confidence regions and point estimates which is called exact since it reduces to Fisher's so-called exact test in the case of the hypothesis of independence in a 2 × 2 contingency table. All the wellknown standard tests based on exact sampling distributions are instances of the exact test in its general form. The likelihood ratio and x₂ tests as well as the maximum likelihood estimate appears as asymptotic approximations to the corresponding exact procedures.     

Exact and best confidence intervals for the ability parameter of the Rasch model

A commonly used method to evaluate the accuracy of a measurement is to provide a confidence interval that contains the parameter of interest with a given high probability. Smallest exact confidence intervals for the ability parameter of the Rasch model are derived and compared to the traditional, asymptotically valid intervals based on the Fisher information. Tables of the exact confidence intervals, termed Clopper-Pearson intervals, can be routinely drawn up by applying a computer program designed by and obtainable from the author. These tables are particularly useful for tests of only moderate lengths where the asymptotic method does not provide valid confidence intervals.     

Stuart’s tau measure of effect size for ordinal variables: Some methodological considerations

The reporting of measures of effect size has become increasingly important in psychology. A Monte Carlo resampling permutation procedure is introduced to find near-optimum maximum values for Stuart’s τ _c measure for two-way ordinal contingency tables, also termed Kendall’s τ _c since Kendall introduced τ _a and τ _b . Comparisons between resampling and exact procedures demonstrate the accuracy and utility of resampling measures of effect size for two-way ordinal contingency tables. The resampling procedure is shown to be more precise than the traditional method of standardizing τ _c .     

Correlations between rats' spatial location and intracranial stimulation administration affects rate of acquisition and asymptotic level of time allocation preference in the open field

The present experiment explores the effects of the response (1-sec occupancy of a target area in an open field)-reinforcer (intracranial stimulation) contingency on time allocation in the open field in rats. The probability of reinforcement given response (X) and the probability of reinforcement given nonresponse (Y) were varied randomly across sessions within a subject. The 21 contingency treatments explored included all possible combinations of values (0, .1, .2, .3, .4, .5) of X and Y such that X ≥ Y. The results indicate that rate of acquisition and asymptotic level of time allocation preference to the target area are negatively related to the value of Y (for any given value of X). Variations in X (for any given value of Y) were less effective. Evaluation of proposed contingency metrics revealed that the Weber fraction (X − Y)/X most closely approximates performance, and that the value of the difference detection threshold derived from the Weber fraction is a constant.     

The effect of contingency upon the appetitive conditioning of free-operant behavior

The problem of maintaining independence between response rates and reinforcement probabilities when determining the effect of varying the response-reinforcement contingency upon free-operant behavior was solved by programming local reinforcement probabilities for response and no response on a second-by-second basis. Fifty-seven rats were trained to lever-press on schedules of water reinforcement involving different values of contingency. All rats were first trained on a high positive contingency and then shifted to less positive, zero, or negative contingencies. Under these conditions, rate of lever-pressing declined appropriately when the contingency between response and reinforcement decreased or was made negative. The decline in rate produced by a zero contingency cannot be attributed to extinction, since the probability of reinforcement given the occurrence of a response was the same as for the positive contingency from which the shift to zero was made. That is, there was no change in the opportunity for response-reinforcement contiguity. It was concluded that the technique of programming local reinforcement probabilities offers promise for more critical examinations of the effects of contingency upon free-operant behavior.     

Techniques for Simultaneous Inference

When more than one significance test is carried out on data from a single experiment, researchers are often concerned with the probability of one or more Type I errors over the entire set of tests. This article considers several methods of exercising control over that probability (the so-called family-wise Type I error rate), provides a schematic that can be used by a researcher to choose among the methods, and discusses applications to contingency tables.