Interobserver Agreement (IOA) is an important step in ensuring data accuracy and reliability. Motivity makes it simple to set up workflows for IOA using Session Types. Below are two recommended approaches.
Option 1: Use a Printed Data Sheet
If you want a quick and easy method:
- Print the corresponding data sheet for the programs via the “Collect Data” view
- Have the secondary observer record data on paper during the session.
- Compare results after the session is complete.
This method is straightforward and works well when IOA is only needed occasionally.
Option 2: Create an IOA Session Type
For teams that want digital IOA collection inside Motivity:
- Clone Programs for IOA
- Clone the program(s) you want to include in the IOA Session Type.
- Clone the program(s) you want to include in the IOA Session Type.
- Rename each cloned program using a consistent naming convention so they’re easy to tell apart from the originals.
- For example: “IOA – Program Name”
- For example: “IOA – Program Name”
- Consider adding an “IOA” label to the cloned program. This will help with organization and make it easier to filter and quickly locate all IOA programs within your learner’s programs.
- Assign each target you would like to take IOA data on to the “IOA Session Type” under the target’s properties. Note: for IOA you can choose to delete any baseline or maintenance phases and take the IOA only in the train phase. If you want to take IOA in all phases, you will need to adjust the properties of each phase.
- Adjust permissions so that RBTs/therapists cannot run or collect data in this session type. This ensures only supervisors or designated observers use it.
- The RBT runs the original program in the standard session type, while the BCBA (or second observer) uses the cloned version in the IOA Session Type.
- This article reviews session types in depth.
Graphing and Analysis
- Motivity allows you to overlay the RBT’s target data and the BCBA’s target data on the same graph. Here is an article showing how to set that up.
- This makes it easy to compare performance, calculate IOA, and analyze agreement across observers.
- This makes it easy to compare performance, calculate IOA, and analyze agreement across observers.
Why This Is Helpful
- Flexibility: You can use either paper or digital methods depending on what fits your workflow.
- Accuracy: Cloning programs ensures that observers are scoring the same targets with the same instructions, measures, etc.
- Visualization: Overlay graphs make it easy to analyze and present IOA data.