1. Start with Your Research Idea

Every study is unique but there are some overarching design features we want to demonstrate in these examples.

  • Stand alone problem sets or using the individual assignment feature (this requires cooperation with a teacher)
  • Skill Builder problem set or a problem set where there is a set number of problems (this, especially the Skill Builder, works well with our subject pool).
  • Responses giving in the "hint" boxes or using the "scaffolding" system or using the feedback message.

What approach should I use to run my study?

There is a single problem set study (most often as a skill builder study) and an orchestrated study. Orchestrated studies are ones where the researcher recruits a few teachers to participate. A single problem set study in particular a skill builder study will get used by many more students because all a teacher needs to do is find it and assign it.

If I use the ASSISTments subject pool, how will teachers find my study?

In this video, Professor Heffernan shows the "ASSISTments Certified Problem Sets" folder, where teachers will be able to find your study. The description and Common Core standard that serve as the title will be followed by "EX" to denote an experimental problem set. In this video, Professor Heffernan also discusses a little bit about the ARRS system and how researchers can benefit when teachers have this preference selected.

Examples

Video vs. Text Feedback

This study was an initial exploration into replacing textual ASSISTments feedback with matched content videos. It is related to Rich Mayer's Multimedia principles.

Example Problem Set

Each student sees a total of 11 problems, 6 of them are Pythagorean Theorem problems.

This problem set starts with a linear section. The ID for this problem set is PSAHVAN – make a copy and take a closer look in your account. There is an introduction about setting the volume on the computer to be ready for the video. Then comes the experiment which has a choose condition. Finally there are 4 survey questions.

There were 4 conditions with different orders of video and text. Note how the Choose Condition is evoked fro this section. The published study used Group 1 and 3 as one condition (video first) and Group 2 and 4 (text first) . The first problem was experiment and the second problem was the assessment of the experiment.

Studying the Dweck Intervention

This study is in preparation. Working with a colleague from the Lytics Lab at Stanford University, we designed this problem set to further investigate the effect of Dweckian motivational messages within ASSISTments. In each condition the students were asked these questions before and after a video to gauge their mindset.

Example Problem Set

This is a stand alone problem set where a skill builder does the randomization. This is a Skill Builder, students have to get three prime factorization problems correct.


The outside problem set is a skill builder (see orange). The ID for this problem set is PSAKUSU in case you want to make a copy and take a closer look. Because it is a skill builder one of the three sections (see purple) is selected at random. Next the student goes through one of these sections in a linear order.

The first section PPTC has two sections first the Intro then the SB, they are given out in linear order (see orange). You can see in the green that this is showing the PPTC section which has no individual problems just two sections.

The intro has three problems, first M1 the survey question then the video and finally M2 the second survey question (see purple). These are delivered linearly then the student goes to the SB section where they work until they get three right in a row. The three problems in the Intro are in "Test" mode and will be marked wrong (a clever combination) this allows these problems to not count for the three right in a row in the skill builder.

Adaptive Homework

This study was done in two parts. In the first part Group A received the experimental feedback, then in the second part Group B received the experiment feedback. In order to do this the researchers had to use individual assignments instead of the choose condition. Here are the two conditions. Below you can see how the problem sets were assigned.

Example Problem Set

These assignments are assigned using the individual assignment feature. Each student sees a total of 12 problems as their homework.

This is the problem set for the experiment. The ID for this problem set is PSAMDQP in case you want to make a copy and take a closer look. It has 12 items these problems come directly from the Connected Math text book as you can see by the question numbers. These ones each just have correctness feedback.

This is the problem set for the experiment. The ID for this problem set is PSAKWXB in case you want to make a copy and take a closer look. It has 12 items these problems come directly from the Connected Math text book as you can see by the question numbers. These ones each have the adaptive feedback.

Worked Examples

The results indicated that tutored problem solving is significantly better than worked examples in terms of the average gain of students in each condition. The scaffolding question tutoring also took significantly more time.

Example Problem Set

A choose condition is used. Each student sees a total of 11 problems, 6 of them are Pythagorean Theorem problem

This is the problem set editor. All the students do the three parts Pre-Test, Experiment, Post-Test (purple). Students do these in a linear order (orange). The next image show what the Experiment looks like.

In the Experiment you can see the choose condition (orange) is used to do the randomization into condition between the Scaffold and the Worked Example condition (purple) in the Experiment section (green).

Here you can see the three items (purple) in the scaffold section (green). These are delivered linear order (orange)