USING A MIDTERM WARNING SYSTEM TO IMPROVE STUDENT PERFORMANCE AND ENGAGEMENT IN AN INTRODUCTORY STATISTICS COURSE: A RANDOMIZED CONTROLLED TRIAL
DOI:
https://doi.org/10.52041/serj.v22i3.406Keywords:
Statistics education research, Student engagement, e-mail nudges, Introductory statistics, Randomized controlled trialAbstract
This article reports on an evaluation the effectiveness of e-mailed grade “nudges” on students’ performance and engagement in an introductory statistics course for undergraduate health science students. In 2020–2021, 358 students were randomized to an e-mail (n = 178) or no e-mail (n = 180) group. The intervention e-mail contained information on each student’s predicted final grade (grade nudge). Using two-sample t-tests, the statistical analysis of final grades in the course showed a higher compatibility with a model of no mean difference for students in the e-mail (73.5%) vs. no e-mail (72.1%) group. Comparison of the distributions of final grades between the two groups, however, suggested the e-mailed nudges may be related to slight improvements in final grades. Specifically, the median final grade was higher in the e-mail group (74.6 vs. 72.4); the Q1 value in the e-mail group was also higher, and the interquartile range was similar: no e-mail group (15.8) vs. e-mail group (14.2). Students also completed the Scale of Student Engagement in Statistics (SSE-S). Total engagement, affective and cognitive subscale scores of the SSE-S were higher in the e-mail group, resulting in low compatibility with a model of no difference in engagement scores. Overall, the results showed there is potential for our midterm warning system to be used to improve outcomes, particularly given that it is simple to implement, cost-effective, and easily scalable.
References
Appleton, J. J., Christenson, S. L., & Furlong, M. J. (2008). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools, 45, 369–386.
Baumgartner, T. A., & Chung, H. (2001). Confidence limits for intraclass reliability coefficients. Measurement in Physical Education and Exercise Science, 5(3), 179–188. https://doi.org/10.1207/S15327841MPEE0503_4
Boretz, E. (2014). Midsemester academic interventions in a student-centered research university. Journal of College Reading and Learning, 42(2), 90–108. https://doi.org/10.1080/10790195.2012.10850356
Butcher, K. F., & Visher, M. G. (2013). The impact of a classroom-based guidance program on student performance in community college math classes. Educational Evaluation and Policy Analysis, 35(3), 298–323.
Bravo, G., & Potvin, L. (1991). Estimating the reliability of continuous measures with Cronbach’s Alpha or the intraclass correlation coefficient: Toward the integration of two traditions. Journal of Clinical Epidemiology, 44(4–5), 381–390.
Carini, R. M., Kuh, G. D., & Klein, S. P. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47(1), 1–32.
Chen, Q., & Okediji, T.O. (2014). Incentive matters! The benefit of reminding students about their academic standing in introductory economics courses. The Journal of Economic Education, 45(1), 11–24.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge Academic.
Crisp, G., Nora, A., & Taggart, A. (2009). Student characteristics, pre-college, college, and environmental factors as predictors of majoring in and earning a STEM degree: An analysis of students attending a hispanic serving institution. American Educational Research Journal, 46(4), 924–942.
Damgaard, M. T., & Nielsen, H. S. (2018). Nudging in education. Economics of Education Review, 64, 313–342.
Deslauriers, L., Harris, S., Lane, E., & Wieman, C. (2012). Transforming the lowest-performing students: An intervention that worked. Journal of College Science Teaching, 41, 80–88.
Fredricks, J. A., & McColskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 763–782). Springer Science + Business Media.
Gordanier, J., Hauk, W., & Sankaran, C. (2019). Early intervention in college classes and improved student outcomes. Economics of Education Review, 72, 23–29. https://doi.org/10.1016/j.econedurev.2019.05.003
Jimerson, S. R., Campos, E., & Greif, J. L. (2003). Toward an understanding of definitions and measures of school engagement and related terms. California School Psychologist, 8, 7–27.
Jones, B. D., Paretti, M. C., Hein, S. F., & Knott, T. W. (2010). An analysis of motivation constructs with first-year engineering students: Relationships among expectancies, values, achievement, and career plans. Journal of Engineering Education, 99(4), 319–336. https://doi.org/10.1002/j.2168-9830.2010.tb01066.x
Kahu, E. R. (2013). Framing student engagement in higher education. Studies in Higher Education, 38(5), 758–773. https://doi.org/10.1080/03075079.2011.598505
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155–163.
Lawton, S., & Taylor, L. (2020). Student perceptions of engagement in an introductory statistics course. Journal of Statistics Education, 28(1), 45–55. https://doi.org/10.1080/10691898.2019.1704201
Marks, H. M. (2000). Student engagement in instructional activity: Patterns in the elementary, middle, and high school years. American Educational Research Journal, 37, 153–184.
Moss, B. G., & Yeaton, W. H. (2015). Failed warnings: Evaluating the impact of academic probation warning letters on student achievement. Evaluation Review, 39(5), 501–524.
Muir, S., Tirlea, L., Elphinstone, B., & Huynh, M. (2020). Promoting classroom engagement through the use of an online student response system: A mixed methods analysis. Journal of Statistics Education, 28(1), 25–31. https://doi.org/10.1080/10691898.2020.1730733
Niranjan, S., Wu, J., & Jenner, C. (2015). Implications of student intervention and antecedents on academic motivation and success. International Journal of Education Research, 10(2), 1–21.
Ober, T. M., Hong, M. R., Rebouças-Ju, D., Carter, M. F., Liu, C., & Cheng, Y. (2021). Linking self-report and process data to performance as measured by different assessment types. Computers & Education, 167. https://doi.org/10.1016/j.compedu.2021.104188
Osborne, J., Parlier, R., & Adams, T. (2019). Assessing impact of academic interventions through student perceptions of academic success. The Learning Assistance Review, 24, 9–26.
Pugatch, T., & Wilson, N. (2018). Nudging study habits: A field experiment on peer tutoring in higher education. Economics of Education Review, 62, 151–161.
Rafi, Z., & Greenland, S. (2020). Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise. BMC Medical Research Methodology, 20. https://doi.org/10.1186/s12874-020-01105-9
Schaufeli, W. B., Bakker, A. B., & Salanova, M. (2006). The measurement of work engagement with a short questionnaire: A cross-national study. Educational and Psychological Measurement, 66, 701–716. https://doi.org/10.1177/0013164405282471
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
Smith, B. O., White, D. R., Kuzyk, P. C., & Tierney, J. E. (2018). Improved grade outcomes with an e-mailed “grade nudge”. The Journal of Economic Education, 49(1), 1–7.
Tai, R. H., Liu, C. Q., Maltese, A. V., & Fan, X. (2006). Planning early for careers in science. Science, 312(5777), 1143–1144.
Thompson, B. (2007). Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychology in the Schools, 44, 423–432. https://doi.org/10.1002/pits.20234
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
Willingham, W. W., Pollack, J. M., & Lewis, C. (2002). Grades and test scores: Accounting for observed differences. Journal of Educational Measurement, 39, 1–37.
Whitney, B. M., Cheng, Y., Brodersen, A. S., & Hong, M. R. (2019). The scale of student engagement in statistics: Development and initial validation. Journal of Psychoeducational Assessment, 37(5), 553–565. https://doi.org/10.1177/0734282918769983
