The Basics of Building
Your study will take the form of one (or more) problem set(s). This page describes how to create problems for your problem set(s). You will also learn how to share your problem sets with your colleagues as you work.
Intro Resources
Student Supports
As a researcher, tutoring strategies will likely become the focus of your experimental design. You can use these methods to help students solve a problem or to deliver an intervention. Please note that if you add more than one strategy to a single problem and both strategies are enabled, students will be randomly assigned to one or the other, they will not see both.
Hints
Hints are simple statements that students can request on demand. Hints do not require students to answer any sub-questions, they just offer guidance. Students can request multiple hints and ultimately opt to "Show Answer," ensuring that they will reach the next problem, or finish the problem set. This (somewhat antiquated) video offers a step-by-step tutorial for adding hints.
Scaffold Problems
Students can request that a problem is broken down into a set of sub-questions. Scaffolded subquestions can be presented on demand or set to open automatically when students get a problem wrong. These (somewhat antiquated) videos explain how to add scaffolds to a problem set: Part 1 and Part 2.
Common Wrong Answer Feedback
When students enter a common wrong answer, they can receive a feedback message specifically tailored to that answer. This type of feedback is presented automatically when students submit a common wrong answer.
As you grow more accustomed to ASSISTments, you can get quite creative with student supports. You may wish to nest a scaffold within a scaffold to better design a condition. Advanced building instructions are available for extra guidance.
Content Labeling
We ask that you add labels to your section or problem names according to the structure of your experimental design to simplify data reporting. Labels follow the name of a section or problem and are placed in square brackets [...]. They must be spelled accurately and appear in all lowercase.
Examples of Existing Labels and their Meanings
NAME OF SECTION OR PROBLEM [pretest]
Signifies the section or problem meant to be analyzed as a pretest.
NAME OF SECTION OR PROBLEM [ignore]
Signifies the section or problem meant to be ignored in analysis; data will still be provided in log files. Most commonly used for problems that convey directions or messages.
NAME OF SECTION OR PROBLEM [control]
Signifies the section or problem meant to be analyzed as the control condition. Can also be numbered in the case of multiple control conditions ([control1], [control2], etc.)
NAME OF SECTION OR PROBLEM [treatment]
Signifies the section or problem meant to be analyzed as the treatment condition. Can also be numbered in the case of multiple treatment conditions ([treatment1], [treatment2], etc.)
NAME OF SECTION OR PROBLEM [posttest]
Signifies the section or problem meant to be analyzed as a posttest.
Externally Run Problems
Documentation on how to create Externally Run Problems are presented below. This problem type allows you to integrate another learning resource with ASSISTments. You can integrate most systems but will only receive data from what occurs within the external system if you host the system or have access to its data. Examples from a student perspective are also presented below.
Try a problem set with BBN as the externally run learning resource
Section Types
There are six section types that control how students receive content. You can access these section types (once you have turned on all beta/research preferences in your account) selecting Input Problem ID's under Create a Problem Set within your Builder Tab. Alternatively, you can edit sections in work that you have already started by finding the problem set(s) you wish to change in your Builder Tab under Problem Sets I've Built and selecting Edit. Each section type is described in greater detail below.
Some sections allow you to Display the Problem Set as if it were a Skill Builder. If you have made a randomized controlled experiment that will be released using Auto Deploy, this setting should be turned on and the original Skill Builder that is being replaced should be entered as the Skill Builder used in ARRS. Remember to Save Changes.
Complete All - Linear Order
This section type mimics a traditional worksheet but requires that problems are solved in the order in which they are compiled in the builder. For instance, if you create 10 problems, students will work until completing all 10. All Students experience problems in the same order (P1, P2, P3, P4, P5, P6, P7, P8, P9, P10). Each problem must be completed before the next problem is shown. This section type is scored using percent correct.
Complete All - Random Order
This section type also mimics a traditional worksheet but the problems that are compiled in the builder will be presented randomly to each student. For instance, if you create 10 problems, students will still work until completing all 10. However, Student A may experience (P2, P5, P3, P8, P9, P1, P4, P7, P6, P10) while Student B may experience (P9, P3, P6, P1, P4, P2, P8, P7, P10, P5). Each problem must be completed before the next problem is shown. This section type is scored using percent correct.
Skill Builder - Linear Order
This section type creates a problem set that requires students get n problems right in a row to achieve mastery and they must be solved in the order in which they are compiled in the builder. For instance, if you create 50 problems, students will work until they accurately solve n problems in a row. All Students experience problems in the same order (P1, P2, P3, P4, P5, P6, P7, P8, P9, P10...) but not all students will see all problems - some may master after Problem 3, others later in the assignment, and others not at all. Each problem must be completed before the next problem is shown. This section type is scored based on mastery.
Skill Builders have settings, described in greater detail below. If you make changes to any of these settings, make sure to Save Variables.
Completion Count
The number of Problems the student must answer correctly in a row to complete the section. ASSISTments only considers answers correct when the student answers the correct response on their first attempt and did not use student supports. The default setting is 3 problems correct in a row.
Test Out Count
Allows a student to complete the Skill Builder by getting a preset number of problems correct in a row at the start of the section. For example, if Test Out Count is set to 1, students only have to answer the first problem correctly to master the skill. However, if they make a mistake or request student supports they will need to get n problems in a row (the Completion Count). The default setting is a Test Out Count of 0.
Day Limit
The maximum number of Problems a student can attempt to answer in a single day, not including being 'on a roll'. For example, if the Day Limit is set to 10, students can try to get n right in a row until they reach the 10th problem - unless they get the 10th problem correct, which will allow them to see additional problems until they either answer incorrectly or reach n right in a row. Thus, it is possible for students to see the number of Problems in the Day Limit plus (n - 1) where n is the Completion Count. For the default of 3 right in a row this means that a student can master the problem set on Problem 12 (correctly answering problems 10, 11, and 12). If students fail to reach mastery and the Day Limit is hit they are told to try again tomorrow. If using a Day Limit, be sure to adjust for being 'on a roll'. The default setting is a Day Limit of 10.
Stop On Day Limit
The section AND problem set stop and students see the Day Limit message if this setting is turned on. If this is turned off, the section stops but the problem set continues. This can be helpful if you want students to move on to a posttest regardless of whether or not they have mastered a Skill Builder used as your treatment/control.
Skill Builder - Random Order
This section type creates a problem set that requires students get n problems right in a row to achieve mastery but the problems that are compiled in the builder will be presented randomly to each student. For instance, if you create 50 problems, students will work until they accurately solve n problems in a row. Student A may experience (P2, P5, P3, P8, P9...etc) while Student B may experience (P9, P3, P6, P1, P4... etc) AND not all students will see all problems - some may master after Problem 3, others later in the assignment, and others not at all. Each problem must be completed before the next problem is shown. This section type is scored based on mastery.
Skill Builders have settings, described in greater detail in the drop down above for Skill Builder - Linear Order. They function the same way when using Random Order. If you make changes to any of these settings, make sure to Save Variables.
Choose One Item or Section - Random Order
This section type randomly assigns a problem or subsection to each study. It can be used to assign students to a treatment/control condition. For example, if you are testing the effectiveness of two types of student supports, you might create a problem set with a Choose Condition housing two sections: Student Support A and Student Support B. When students launch the assignment they will be randomly assigned to either the content in section Student Support A or that in section Student Support B. These subsections might be Complete All sections, assessed by percent correct, or Skill Builder sections, assessed by mastery. This section type alters teacher reports.
If-Then-Else
The If-Then-Else section type requires you to add 2-3 objects:
Conditional Statement
This can be a Complete All section, a Skill Builder section, or a single problem. The conditional returns a true or false statement that determines what students see next.
A Complete All section will return true if the percent correct is greater than the set Problem Set Correctness variable.
A Skill Builder will return true if the student reaches mastery and false if all problems are exhausted.
A Problem will be true if the student gets it right and false if it is wrong.
Use of a Skill Builder or a single Problem do not require Problem Set Correctness to be altered. The Problem Set Correctness can be altered if using a Complete All section. If changes are made to any settings, remember to Save Variables.
Then
What students receive if the Conditional Statement returns true. This can be empty if the If-Then-Else is not nested within a Choose Condition, simply toggle the appropriate setting and Save Variables.
Else
What students receive if the Conditional Statement returns false. This can be empty if the If-Then-Else is not nested within a Choose Condition, simply toggle the appropriate setting and Save Variables.
Notes on If-Then-Else Section Types
If the Conditional Statement is a single Problem, the problem may be more than a simple right/wrong problem.
If it has multiple main problems it will return the average correctness across the main problems. The only way for this to return "true" is for a student to get all main problems correct.
Scaffold questions are ignored. Although scaffolds are logged as additional problems in our database, students using scaffolds will get the problem wrong (or receive partial credit) because they have used a student support. As such this will return "false".
Ungraded Open Response problems are considered incorrect by our database. As such, they will always return "false".
Examples of the If-Then-Else
Multimedia Verification
If your study involves audio or video, you can assign a Problem that allows students to verify that they are able to access the media type. Some schools ban YouTube. This type of verification allows you to confirm that only students who can access all types of content potentially displayed in your study actually participate. Those who cannot access the media type (i.e., video if a school has blocked YouTube) can be routed into an alternative assignment following business as usual.
Adaptive Content Delivery
You can assign a Problem that gauges student skill as the Conditional Statement and assign a prerequisite if students perform poorly. If students perform well you can provide a message or skip to another section entirely.
Advanced Building
Example studies
On our Example Studies page, we describe four studies that have been developed using the legacy version of ASSISTments:
Comparing Video and Text Feedback
Assessing the Effects of Grit and Mindset
Assessing Adaptive Homework
Assessing the Effects of Worked Examples
For more information about how to build problem sets with more complicated structures, visit our Advanced Builder Site