This paper compares the type I error rate and computes the kappa for agreement on significance decisions using a new application of sampled permutation tests and log linear analysis to examine whether two sequential associations are different within a single dyad (e.g., a teacher and a student). We used the sample permutation test and log linear analysis to test the significance of the same pairs of behavior streams. Within each set of behavior stream pairs, the behavior streams were generated from the same algorithm to create a distribution in which the mean difference between sequential associations was zero. Using generated behavior streams allows us to estimate the type I error rate and to create the sampling distribution needed to understand the conditions under which the techniques provide different results. However, the algorithm is based on an actual mother-child interaction session that yielded moderate positive sequential association. Because past work has found that permutation and asymptotic tests produce different results most often when sample sizes are small, we generated sets of behavior pairs with relatively small number of events. Each member of a pair had the same (i.e., 100 vs. 100, 50 vs. 50, and 25 vs. 25) and different (100 vs. 400) number of event pairs to simulate designs that use time-based (typically producing equal-length comparisons) and event-based (typically producing different-length comparisons). In the 100 vs. 100 event pairs condition, the amount of agreement on significance decisions was extremely high (kappa = .92) and the type I error rate was about approximately the same for both methods (.059, .066 for shuffle-the-cell and log linear, respectively). In the 50 vs. 50 and the 25 vs. 25 event pairs conditions, the degree of agreement reduced and the difference in type I error rates increased as length of session decreased. In all equal-length cases, the shuffle-the-cell method was slightly more conservative than the log linear method. In the different length condition, the type I error rates for the two methods (.044 and .061, for shuffle-the-cell and log linear, respectively) were similar but there was almost no agreement above that expected by chance regarding significance decisions (kappa = .07). Cases in which the results differ are those that violate the assumption that the data are sampled from a Poisson distribution. It appears that when sample sizes are small and the behavior streams to be compared are of very different lengths, permutation tests may be preferable over log-linear analysis, at least for comparing 2 x 2 tables.

© 2000 Noldus Information Technology b.v.