Opportunity Outcome

This article describes how to configure the Opportunity Outcome in the Data Collection section of a target phase in the Program Builder.

 

 

This article has three parts:

• What is an Opportunity Outcome
• Opportunity Score Calculation
• Opportunity Success Criteria

What is an Opportunity Outcome 

As described in the Introduction to the Program Builder, an Opportunity corresponds to a single "exercising" of a target, and is the basic container for all data in Motivity, e.g. the performance of a discrete trial, the execution of a chain of steps in a task analysis, or a period of behavior tracking.  The "Opportunity Outcome" is the quantitative score yielded by that Opportunity when data is collected, and the qualitative designation as to whether the Opportunity has succeeded or failed (or neither).  

Opportunity Score (Quantitative Outcome)

Each Opportunity yields a numeric score when data is collected. If a target has a single measure, then the Opportunity Score is equal to the measure score.  If the target has more than one measure, the Opportunity Score is usually the average or sum of the measure scores, as determined by the target's Opportunity Score Calculation method.

As a simple example, consider a discrete trial target with a single Pass-Fail measure.   The Opportunity Score would be equal to the measure score: 100 if + was pressed, and 0 if - was pressed.  During data collection, the Opportunity Score is displayed to the right of the Session Outcome:

The Opportunity Score is important because 1) it is used in determining if an Opportunity succeeds or fails, and 2) it is sometimes used in calculating the Session Score (which gets plotted on performance charts). For example, when tracking the learner's performance on a daily quiz in a workbook, the Opportunity Score would be the quiz score, which in turn could be used as the Session score, to plot the numeric performance over time rather than just an all-or-nothing indicator of Succeeded or Failed.  

Success Designation (Qualitative Outcome)

Every Opportunity is considered a Success, Failure, or Indeterminate (neither success nor failure).  During data collection, the qualitative Outcomes are conveyed in the colors of the Opportunity Indicators (small squares) in the bar under the target name.

The designation as to whether an Opportunity is a failure, success, or indeterminate is determined by applying the Opportunity Success Criteria to the score of that Opportunity, as described below.  For example, a discrete trial target's Opportunity Success Criteria is typically configured to succeed if the score is 100, or otherwise fail.  

Opportunity Outcome vs. Session Outcome

It is important to distinguish between Opportunity Outcome and the Session Outcome: the former refers to the result of a single performance of a target, whereas the latter refers to the overall performance of that target during the Session, as determined by evaluating all the Opportunity Outcomes for the target.  In the example above, there are five Opportunity Outcomes: 100 (Success), 0 (Failure), 100 (Success), 100 (Success), and 0 (Failure).  The Session Outcome is 60% (Failure), depicted by the pale red background of the bar behind the Opportunity Indicators.  See the article on Session Outcome for more information about how Session Outcomes are computed.

Opportunity Score Calculation

The Opportunity Score ("Opp. Score") is a number whose value is determined by the Instrument readings.  Instrument scores are as follows:

Instrument	Instrument Score
Count	the final count value when the opportunity ends
Countdown Timer	the amount of time remaining on the timer when the opportunity ends (in seconds)
Custom Prompts	the score associated with the button pressed
Duration	the measured duration in seconds
Integer	the integer that was entered
Pass-Fail	0 (-) or 100 (+)
Text	None

 

Instrument Instrument Score

The only Instruments which participate in the Opp. Score calculation are the ones which are marked as "Affects Outcome" in the Instrument definition in the Target Interface:

If there is a single Instrument affecting outcome, then the Opportunity Score is simply equal to that Instrument's score.  However, if there are multiple Instruments affecting outcome (e.g. a Task Analysis target with a Custom Prompt Instrument for each step), then the Opportunity Score calculation is determined by the "Opp. Score" setting.

Automatic

The Automatic setting will always use the Average of instrument scores, except when Opportunity Success Criteria is in "Advanced" mode.  In that case, the score calculation method depends on the rule that is fired.  This will be explained in more detail below. 

Measure Scores

If the Automatic setting is not yielding the desired results, you can explicitly control how the score should be calculated by setting it to Measure Scores:

This allows you to choose between Average, Sum, Minimum, or Maximum.

For example, to implement a PEAK scoring system, use Sum instead of Average:

Opportunity Success Criteria

The Opportunity Success Criteria specifies how to determine if an Opportunity is a Success, Failure, or Indeterminate. 

There are three modes: None, Basic, and Advanced.

Mode: None

When the mode is set to None, then the Opportunity never succeeds nor fails, and is always indeterminate. This is the default mode when creating a new Behavior Target.

Mode: Basic

When the mode is set to Basic, then the Opportunity succeeds if ALL its Instruments meet a specified minimum threshold, otherwise it fails.  This is the default mode when creating a new Skill Target. The default threshold is set to 100 (which corresponds to the score of the + button in Pass-Fail).  

Sometimes you will want to change the threshold.  For example, if you are implementing a Prompt Hierarchy with a Custom Prompts Instrument using default button scores, a threshold of 100 means that only the right-most button would be designated a success:

The default button scores for a five-button prompt are 0, 25, 50, 75, and 100.  So in this example, if you want Verbal prompt (V) to be considered a success, you can change the threshold to 75:

In the event that you have two or more Instruments, Basic Mode requires that all of them exceed the threshold for the Opportunity to be considered a success.  For example, in a Task Analysis with multiple steps, Basic Mode will require that all steps succeed for the task to be considered a success:

 

Mode: Advanced

Sometimes Basic mode is not expressive enough to achieve the desired results.  For example, in the task analysis target shown above, you may want to relax the criteria so that an Opportunity is a success if all but 1 of the steps succeed, rather than requiring all of them to succeed.  To achieve this, you can switch to Advanced mode and create a rule as follows:

As you can see, Advanced Mode lets you define any number of rules, each of which specifies a condition for designating a qualitative Outcome. The rules are processed in the order in which they appear in the list: the first rule is evaluated to determine if its condition is met; if it is, that rule "fires", dictating the Outcome, and the processing stops. If it's condition is not met, the second rule is evaluated, etc. If none of the rules fire, then the Outcome is determined by the fall-through "Otherwise" clause.

Each rule has several parts. The first part is the Outcome designation clause (succeeds, fails, or indeterminate) that dictates what happens when the rule fires:

The next part of the rule depends on whether or not the target has multiple measures.  If there are multiple measures, there will be an Instrument aggregate clause, which lets you specify which of the measures are to be considered in determining if the rule condition is met. Note: if there is only one measure, this clause will not appear.

This provides a flexible and powerful mechanism for defining rules based on the average, sum, minimum, or maximum of the Instrument scores, or requiring that all or some subset (e.g. "any 3", or "50%") of the scores individually meet the criteria. 

The final part is the Score matching clause which is a numeric predicate (e.g. "at least 100" or "between 50 and 75") that is applied to the aggregate value or range of scores defined in the Instrument aggregate clause:

The Score matching clause also has an editable numeric threshold that you can edit (e.g. "100"), or two thresholds when "between X and Y" is chosen. Together, the Instrument aggregate clause and the Score matching clause create a logical condition (e.g. "all but 1 of the instrument scores are at least 100"), which evaluates to either true or false. If it evaluates to true, and no other rule has yet fired, then the rule fires, and the Outcome designation clause is applied.

As an example, consider a Reading Fluency target in which the learner is given 5 minutes to read a passage aloud. The target will be considered a success if the child reads at least 140 words, without making 3 or more errors. The Target Interface can be defined as a Timer (configured to not affect outcome), an Integer to enter the number of words read, and a button to be pressed if the child makes 3 or more errors:

Assuming the "made 3+ errors" is a Custom Prompt instrument with one button that has a score of 0, then the Opportunity Success Criteria can be defined as follows:

The first rule handles the "made 3+ errors" Instrument-- if that button is pressed then the Instrument will yield a score of zero, causing the Opportunity to fail regardless of the number of words read.  If the button is not pressed, then the 1st rule will not fire, and causing the 2nd rule to be evaluated: if the Integer Instrument yields a value higher than 140, then the Opportunity will succeed.

Note that the order of the rules matters in this case. If the "fail if min score is 0" rule was not first, then it would not override the success rule when the "made 3+ errors" button was pressed. Rules can be re-ordered, cloned, or deleted using the 3-dot menu next to each rule:

Note that for this example to properly display the number of words read in the progress chart, the Opportunity Score Calculation is set to "Measure Score Aggregate: Maximum", to make the Score equal to the "number of words" Instrument (156 in the example above), as the "Automatic" setting would have yielded unexpected results.  

Automatic Score Calculation in Advanced Mode

In the Opportunity Score Calculation section, it was explained that the Automatic setting yields an average score when Success Criteria Mode is None or Basic.   When the Mode is Advanced, the Automatic setting yields a score calculation method which is dependent on the Instrument aggregate clause of the rule that fires, as follows:  

Instrument Aggregate Clause	Score Calculation Method
Average	average of all Instruments
Minimum	minimum Instrument
Maximum	maximum Instrument
Sum	sum of all Instruments
All	average of all Instruments
All but N, Any N, At least N%	average of the Instruments that participate in causing the rule to fire, ignoring the rest