Completed Studies

E-TRIALS has been used in numerous published randomized controlled experiments to investigate methods to improve student learning

Selected works are provided below

ASSISTments has been used in over 27 published randomized controlled experiments to investigate different ways to improve student learning.

13 were completed with 17 other universities, 8 Heffernan in collaboration with others, and 5 without Heffernan as an author.

Studies Without Heffernan Authorship

1. Andres-Bray, M., Hutt, S., Zhou, Y, & Baker K. (submitted). A Comparison of Hints vs. Scaffolding in a MOOC with Adult Learners. Submitted to AIED 2021. Retrieved from https://drive.google.com/file/d/1X9gdtSX0RJdOi5AiR5H5rzSIOs3ovPMl/view?usp=sharing

2. Duquennois, C. (2019). Fictional Money, Real Costs: Impacts of Financial Salience on Disadvantaged Students. (Revise and Resubmit at American Economic Review). Dr. Duquennois has told Neil that this looks like they are close to publishing this journal article. See her website for the status.

3. Fyfe, E. (2016). Providing feedback on computer-based algebra homework in middle-school classrooms. Computers in Human Behavior 63: 568-574. Retrieved from https://drive.google.com/open?id=0B2X0QD6q79ZJbzE2R3RNRmZfZ2c

4. Harrison, A., Smith, H., Hulse, T., & Ottmar, E. (2020). Spacing out!: Manipulating Spatial Features in Mathematical Expressions Affects Performance. Journal of Numerical Cognition. 6 (2): 186-203. DOI: 10.5964/jnc.v6i2.243

5. Jiang, Y., Almeda, M. V., Kai, S., Baker, R. S., Ostrow, K., Inventado, P. S., & Scupelli, P. (2020). Single Template vs. Multiple Templates: Examining the Effects of Problem Format on Performance. In Gresalfi, M. & Horn, I. S. (Eds.), The Interdisciplinarity of the Learning Sciences, 14th International Conference of the Learning Sciences (ICLS) 2020, Volume 2 (pp. 1015-1022). Nashville, Tennessee: International Society of the Learning Sciences. https://repository.isls.org/bitstream/1/6288/1/1015-1022.pdf

6. Koedinger, K. & McLaughlin, E. (2016) Closing the Loop with Quantitative Cognitive Task Analysis. In Barnes, Chi & Feng (eds) The 9th International Conference on Educational Data Mining. Pp 412-417. Retrieved from http://www.educationaldatamining.org/EDM2016/proceedings/paper_152.pdf

7. McGuire, P., Tu, S., Logue., M., Mason, C., Ostrow, K. (2017) Counterintuitive effects of online feedback in middle school math: results from a randomized controlled trial in ASSISTments. Educational Media International. 54:3, 231-244, DOI: 10.1080/09523987.2017.1384161. Retrieved from https://www.tandfonline.com/doi/pdf/10.1080/09523987.2017.1384161

8. Smith, H., Harrison, A., Chan, J. C., & Ottmar, E. (2020). Dynamic vs. static: Which worked examples work best? Poster submission to the 2020 meeting of The Mathematical Cognition and Learning Society. [pre-registration]

9. Walkington, C., Clinton, V., & Sparks, A. (accepted pending minor revisions). The effect of language modification of mathematics story problems on problem-solving in online homework. Instructional Science.

10. Hurst, M. A., Cordes, S. (2018) Labeling Common and Uncommon Fractions Across Notation and Education. Proceeding of Cognitive Science. 1841-1846 Retrieved from http://mindmodeling.org/cogsci2018/papers/0354/index.html

Studies Comparing Feedback to "Business as Usual"

• • Kelly, K., Heffernan, N., Heffernan, C., Goldman, S., Pellegrino, G. & Soffer, D. (2013). Estimating the Effect of Web-Based Homework. In Lane, Yacef, Motow & Pavlik (Eds) The Artificial Intelligence in Education Conference. Springer-Verlag. pp. 824-827. With University of Illinois.

  • Kehrer, P., Kelly, K. & Heffernan, N. (2013). Does Immediate Feedback While Doing Homework Improve Learning. In Boonthum-Denecke, Youngblood (Eds) Proceedings of the Twenty-Sixth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2013, St. Pete Beach, Florida. May 22-24, 2013. AAAI Press 2013. p 542-545.

  • Mendicino, M., Razzaq, L. & Heffernan, N. T. (2009). Improving Learning from Homework Using Intelligent Tutoring Systems. Journal of Research on Technology in Education (JRTE). 41(3), 331-346. With West Virginia University.

Studies Comparing Types of Feedback

• • Singh, R., Saleem, M., Pradhan, P., Heffernan, C., Heffernan, N., Razzaq, L. Dailey, M. O'Connor, C. & Mulchay, C. (2011). Feedback during Web-Based Homework: The Role of Hints In Biswas et al (Eds) Proceedings of the Artificial Intelligence in Education Conference 2011. pp. 328–336.

  • Razzaq, L., Heffernan, N. T., Lindeman, R. W. (2007). What Level of Tutor Interaction is Best? In Luckin & Koedinger (Eds.) Proceedings of the 13th Conference on Artificial Intelligence in Education. IOS Press. pp 222-229.

  • Razzaq, L. & Heffernan, N.T. (2006). Scaffolding vs. hints in the Assistment system. In Ikeda, Ashley & Chan (Eds.). Proceedings of the Eight International Conference on Intelligent Tutoring Systems. Springer-Verlag: Berlin. pp. 635-644.

  • Razzaq, L. & Heffernan, N. (2010). Hints: Is It Better to Give or Wait to be Asked? In Aleven, V., Kay, J & Mostow, J. (Eds) Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS2010) Part 1. Springer. Pages 349-358.

  • Shrestha, P., Wei, X., Maharjan, A., Razzaq, L., Heffernan, N.T., & Heffernan, C., (2009). Are Worked Examples an Effective Feedback Mechanism During Problem Solving? In N. A. Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp. 1294-1299). Austin, TX: Cognitive Science Society.

  • Kim, R, Weitz, R., Heffernan, N. & Krach, N. (2009). Tutored Problem Solving vs. “Pure”: Worked Examples In N. A. Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp. 3121-3126). Austin, TX: Cognitive Science Society. With Seton Hall University, Carnegie Mellon and Albert Einstein College of Medicine.

  • Pardos, Z., Dailey, M. & Heffernan, N. (2011). Learning what works in ITS from non-traditional randomized controlled trial data. The International Journal of Artificial Intelligence in Education. 21, 47-63.

  • Razzaq, L. & Heffernan, N. (2009). To Tutor or Not to Tutor: That is the Question. In Dimitrova, Mizoguchi, du Boulay & Graesser (Eds.) Proceedings of the 2009 Artificial Intelligence in Education Conference. IOS Press. pp. 457-464.

Studies Comparing Socio-Emotional Interventions

• • Kelly, K., Heffernan, N., D'Mello, S., Namias, J., & Strain, A. (2013). Adding Teacher-Created Motivational Video to an ITS. Florida Artificial Intelligence Research Society (FLAIRS 2013). pp. 503-508. With University of Notre Dame.

  • Ostrow, K. & Heffernan, N. T. (2014). Testing the Multimedia Principle in the Real World: A Comparison of Video vs. Text Feedback in Authentic Middle School Math Assignments. The International Educational Data Mining Conference. Retrieved March 10, 2014, from http://tiny.cc/ostrow2014-EDM.

  • Ostrow, K. & Heffernan, N. T. (2015) The Role of Student Choice Within Adaptive Tutoring. In Conati, Heffernan, Mitrovic & Verdejo (Eds) The 17th Proceedings of the Conference on Artificial Intelligence in Education, Madrid, Spain. Springer. pp. 752-755.

Studies Assessing Parental Intervention

• • Broderick, Z., O’Connor, C., Mulcahy, C., Heffernan, N. & Heffernan, C. (2011). Increasing Parent Engagement in Student Learning Using an Intelligent Tutoring System. Journal of Interactive Learning Research, 22(4), 523-550. Chesapeake, VA: AACE. (Longer Version).

Assessing the Automatic Reassessment and Relearning System (ARRS)

• • Soffer, D., Das, V., Pellegrino, G., Goldman, S., Heffernan, N., Heffernan, C.,& Dietz, K. (2014) Improving Long-term Retention of Mathematical Knowledge through Automatic Reassessment and Relearning. American Educational Research Association (AERA 2014) Conference. Division C - Learning and Instruction / Section 1c: Mathematics. PDF, Poster Nominated for the best poster of the session. With University of Illinois.

  • Soffer-Goldstein, D., Pellegrino, J., Goldman, S., Stoelinga, T., Heffernan, N., & Heffernan, C. (submitted) The Effect of Automatic Reassessment and Relearning on the Retention of Mathematical Knowledge and Skills. Submitted to Journal of Applied Research in Memory and Cognition (JARMAC). Elsevier. University of Illinois

  • Xiong, X. & Beck, J. (2014) A Study of Exploring Different Schedules of Spacing and Retrieval Interval on Mathematics Skills in ITS Environment. In Stefan Trausan-Matu, et al. (Eds) The Proceedings of the International Conference on Intelligent Tutoring 2014. LNCS 8474. Pages 504-509.

  • Xiong, X., Wang, Y., & Beck, J. B. (2015, March). Improving students' long-term retention performance: a study on personalized retention schedules. InProceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 325-329). ACM.

Studies Comparing Instructional Approaches

• • Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society. (pp. 471-476.) Austin, TX: Cognitive Science Society. By Carnegie Mellon University.

  • Sao Pedro, M., Gobert, J., Heffernan, N. & Beck, J. (2009). In N.A. Taathen & H. van Rjin (Eds.) Comparing Pedagogical Approaches for Teaching the Control of Variables Strategy. Proceedings of the 31st Annual Conference of the Cognitive Science Society Austin, TX: Cognitive Science Society.

  • Ostrow, K., Heffernan, N.T. & Heffernan, C. (2015) Blocking vs., Interleaving: A Conceptual Replication Examining Single-Session Effects within Middle School Math Homework. In Conati, Heffernan, Mitrovic & Verdejo (Eds) The 17th Proceedings of the Conference on Artificial Intelligence in Education, Madrid, Spain. Springer. pp. 388-347 (acceptance rate 28%). With Carleton College.

  • Lang, C., Heffernan, N., Ostrow, K., & Wang, Y. (2015). The Impact of Incorporating Student Confidence Items into an Intelligent Tutor: A Randomized Controlled Trial. In Santos, Boticario, Romero, Pechenizkiy, Merceron, Mitros, Luna, Mihaescu, Moreno, Hershkovitz, Ventura, & Desmarais (eds.) Proceedings of the 8th International Conference on Educational Data Mining (EDM 2015). Madrid, Spain. June 26-29. pp. 144-149.

Links to an older website show the experimental desings.