Partial Credit

One the the benefits of the E-TRIALS Testbed is that you can learn from more than just looking at Boolean correctness on your posttest to a more nuanced view.

The E-TRIALS Testbed team has done a lot of work showing that this gives you more value. In 2009 we published we can better predict their state test scores if we take into account the number of attempts and the number of hints they take compared to a "Boolean correctness only" attitude.

• Feng, M., Heffernan, N.T., & Koedinger, K.R. (2009). Addressing the assessment challenge in an Intelligent Tutoring System that tutors as it assesses. The Journal of User Modeling and User-Adapted Interaction.19, 243-266. Best Paper of the Year at the Journal

In 2013 we used partial credit per question to better predict the next question in assistments. For instance, in a boolean correctness world, a kid that makes a wrong attempt (as their first action) versus a kid that asks for a hint will both be marked wrong. A boolean correctness way of looking at the world will treat these two kids the same. Furthermore, the kid that made a wrong attempt but then on the 2nd action got the problem correct is a very different student than a student that make 4 more guesses and then asked for every hint. The paper below shows that we can use this intuition to better predict their performance on the next question.

• Wang, Y. & Heffernan, N. (2013). Extending Knowledge Tracing to allow Partial Credit: Using Continuous versus Binary Nodes. In Lane, Yacef, Motow & Pavlik (Eds) The Artificial Intelligence in Education Conference. Springer-Verlag. pp. 181-188.

Others in my team have used this extra information to make further improvements.

• Van Inwegen, E., Adjei, S., Wang, Y., & Heffernan, N.T. (2015) Using Partial Credit and Response History to Model User Knowledge. In the Proceedings of the 8th International Conference on Educational Data Mining EDM2015, Madrid, Spain. ISBN: 978-84-606-9425-0 pp 313-319. (acceptance rate 36%)

• Ostrow, K., Donnelly, C. Adjei, S. & Heffernan, N. T. (2015) Improving Student Modeling Through Partial Credit and Problem Difficulty. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (L@S 2015). ACM, New York doi 10.1145/2724660.2724667 pp 11-20. (acceptance rate 25%)

• Hawkins, W., Heffernan, N., Wang, Y. & Baker, S,J,d.. (2013). Extending the Assistance Model: Analyzing the Use of Assistance over Time. In S. D'Mello, R. Calvo, & A. Olney (Eds.) Proceedings of the 6th International Conference on Educational Data Mining (EDM2013). Memphis, TN. pp. 59-66.

• Duong, H., Zhu, L., Wang,Y. and Heffernan, N. (2013). A prediction model that uses the sequence of attempts and hints to better predict knowledge: "Better to attempt the problem first, rather than ask for a hint". In S. D'Mello, R. Calvo, & A. Olney (Eds.) Proceedings of the 6th International Conference on Educational Data Mining (EDM2013). Memphis, TN. pp. 316-317.

A down side of the above work is we were just predict performance on the next question but can we use this info to better predict something later? In the below paper also used these extra information to predict a test 6 months later

• Wang, Y. & Heffernan, N. (2014). The Effect of Automatic Reassessment and Relearning on Assessing Student Long-term Knowledge in Mathematics. In Stefan Trausan-Matu, et al. (Eds) International Conference on Intelligent Tutoring 2014. pp 490-495. LNCS 8474. (42% acceptance rate)

We have also shown that you can use response times as a useful predict of student performance.

• Wang, Y. & Heffernan, N. (2012). Leveraging First Response Time into the Knowledge Tracing Model. 5th International Conference on Educational Data Mining. pp. 176-179.

So we encourage you, the researcher, to use this extra information to get a more nuanced look at student performance.

Of course we are not the only to do this, or even the first. Even researchers at the venerable Education Testing Service have know for a long time that they can get a more sensitive measure of student if we do this, but ETS is a testing company not meant to help student learn, so they don't bother. See the above papers for many other references to literature for others that have been doing partial credit for a long time.