OVERVIEW

We designed a developmental math dashboard that helps instructors assess classwide learning and student progress.

Snapshot

Role

• Product Designer

Tools & Tech

• Figma
• InVision
• Zeplin

Project Duration

• 8 Months

Methods

Strategy

• Design Thinking
• Info Architecture
• Site Mapping

Research

• Co-Design Collabs
• Customer Interviews
• Usability Testing

Design

• Rough Sketching
• Wireframing
• Rapid Prototyping
  

Brief

In most math courses, concepts are taught to the average student profile - there is no personalization of instruction, and everyone is given the same instruction regardless of their level of comprehension. We aimed to measure college-level remedial math students' proficiency to objectives and develop a personalized learning plan and assessment.

Challenge

Code-name Project RIO is a developmental mathematics product that helps 2-year college students advance math skills using adaptive technology. One of the main focuses from a design process perspective was the working triad between Product, Dev, and Design. This project was the first time we've worked this closely as partners in early design stages.

Focus

This learning experience is a new product for developmental math students that will deliver the highest level of personalized content. Students control the pace at which they learn based on their answers to specific content. Ultimately, students feel more confident by focusing only on topics they don’t know and comprehending the material stress-free.

OUTCOME HIGHLIGHTS

Teammates

• Emily Schneider
• Robert Newell
• Ben Piercy
• Bryttni Wolfe

Educators interviewed

73

User research studies

9

Avg. TAM Rating

4.4

THE AIM

Part 1: Purpose

What was the problem we were trying to solve?

BUILDING AN MVP BRICK-BY-BRICK

Personalized & Direct

Educators have access to real-time insights, improving the classroom experience with clear guidance to support individual students where needed.

DIVING DEEP FOR INSIGHTS

Learners experience a personalized instruction journey, starting with a pre-assessment. Course content is tailored to what each student needs to work on to master a concept or skill. Student performance dictates the adaptive practice within each module and provides optimal support for each individual.

RIO introduces advanced instructor performance reporting and provides at-risk learner insights so instructors can support at-risk students before it’s too late. Instructors are in control by setting a distinct Proficiency Threshold.

Understanding obstacles our customers encounter

From day one, we worked closely with instructors to to mold an intuitive performance dashboard.

RESHAPING THE PROBLEM

The learning experience addresses personalized content, recognizing where students excel and where they struggle, remediating students at the exact moment, and in the precise method, they need - accelerating proficiency.

Product innovation and systems thinking approaches helped advance personalized, adaptive learning systems that scale globally to improve access and learning outcomes. Working toward a single technology ecosystem and single user identity will provide a unified, connected view of customers' needs.

Always building the foundation of testable MVP

Slowing down and talking with developmental math instructors to understand the hurdles

Micro-interactions that take place as a student works through a problem will ensure students get the right help at the right time, to foster motivation and success.

This visual is an early view of the high-level flow to ensure learning outcomes are achieved with adaptive content.

PARTICULAR ESSENTIALS

A course pre-assessment gives insight into which topics, precisely, a student has already mastered, and which topics they still need to learn. Based on the pre-assessment, content is then tailored to each student. As students review streamlined instruction for each objective they need, practice adapts to each student's level of understanding, providing scaffolding for those who need it while fading learning aids for students to demonstrate true understanding.

RIO (working title) is a developmental math courseware currently in development, designed to help students progress through dev math content efficiently and successfully so they can move on to credit-bearing math courses. The tool understands where students excel and where they struggle, delivering a personalized learning path for each student with adaptive practice.

SHARPENING OUR FOCUS

The learning tool will provide streamlined personalized instruction and practice through software - so will allow for acceleration, and will fit in any course format where instructors want students both learning and practicing.

Product Differentiators:

• Adaptive diagnostic
• Personalized learning experience
• Data insights for students and instructors

Dev math adaptive pre-assessment, focus on reporting results at student & class level. RIO aims to help faculty improve developmental math retention and completion rates. 60% of community college students are referred to developmental math, yet only a third of those students go on to finish.

THE WORK

Part 2: Process

How did we arrive at the solution to the original problem?

SWIFTLY ITERATING & GATHERING INSIGHTS

Iterating with feedback

We worked closely with our users to create useful solutions by approaching problems through the lens of an educator and an outcome-focused mindset.

EXPECTATIONS VACILLATE

We kicked the project off with a co-design session with community college developmental math instructors from Colorado. We sat down to better understand what impeded the teaching experience from both a class setup process and paper-work stand point. From their perspective, how students learn best in the modern classroom. These sessions were split into back-and-forth interview questions and simple design tasks to solve their biggest issues.

Diagnostic Co-design findings:

• Start with a temperature check and set the benchmark

• Understand what to spend time on and when in class

• Identify low performers so they can intervene earlier

• Group high and low performers so they can help each other

• Review with students and create individualized study plans

CONNECTING VITAL DOTS

Prototyping with initial designs

The co-design sessions focused on lower-fidelity prototypes to ensure that instructors felt safe to collaborate and share things that seemed wild.

AIMING FOR SWIFT IMPROVEMENT

The research sessions opened w/ a 40-minute interview that informed the co-design exercise where I would build on the screen what instructors wanted to see in a performance dashboard. This helped us better understand what truly mattered to assess the class quickly. We kept the back-and-forth fluid to create an open space for rapid validation that included correction from the instructor and on-the-fly iterations based on what they were seeing. Co-design findings:

• Pace:
  • Instructors want to know if students are on track?
  • Is anyone ready to take a test?
    
• Student-focused landscape assessment:
  • What have they succeeded at since the last class?
    
  • Completion & progress toward a milestone
    • Who needs a nudge?
    • Who needs help?
      

Developing the experience with ongoing validation

Taking the time to fully understand the essential problems instructors face daily

WORKING CLOSELY W/ PARTNERS

We presented instructors with details about students who are struggling with specific content to cover in class or students who aren't completing designated work. Letting users know what is going on and presenting useful information.

These navigation and data point presentation options are good examples of collaboration with users before presenting multiple scale options to partners across product and engineering to ensure feasibility that might include possible API contracts for data point delivery from the adaptive system.

It was important to make sure that the information and data made clear and simple sense to instructors when assessing what the student content needs to brush up on. We worked with product and dev leads to grasp timelines for a simple MVP option that we’ve validated with instructors. There was also improvement potential as we iterated and improved upon the initial release.

RECOGNIZING THE CONNECTION POINTS

Identifying instructors true needs and flows.

We find any open-ended feedback session interesting because it reinforces how pressed for time these instructors truly are. The majority of instructors want some level of control, but don't have time to configure an endless page of settings they don't understand. They also want insights and actions specifically spelled out for them with the ability to dive deeper for more background information.

73

Educators interviewed

9

User research studies

4.4

Avg. TAM rating

PROGRESSIVELY REVEALING WHAT MATTERS MOST TO INSTRUCTORS

Inclusive thinking from Day 1

Endless collaboration

On-the-fly feedback loop

We further researched initial designs with instructors through one-click tests and task completion flows along with design element preference split tests at the end of sessions.

Telling a story w/ data

Pointing out the essentials

These sessions were unmoderated and it's always interesting to hear the direct feedback from instructors when they open up and feel safe talking through their thought process out loud. 

Striving for shared understanding across teams

Stepping outside the design studio and thinking beyond the org's tech stack

It's essential to be thoughtful about interfacing with collaborators at the right moments in the design process:

Gaining direction and product shape visions from stakeholders at the right points. Include engineering and product for feasibility spot checks early and often.

Collaboration went beyond good teamwork to deliver distinctly innovative solutions.

SURFACING THE THINGS THAT MATTER MOST

Actionable flags

—

Through iterative testing it became clear that multiple entry points to specific information was important. We didn’t get this right out of the gate, as instructors sometimes wanted to dig into the deep information from many different points in the experience when testing with task completion.

EXPLAINING HEAVY CONCEPTS WHEN NEEDED

Ongoing help

—

Diving into the details, I found through rapid prototyping that instructors need to understand the class-wide basics:
- What student is this? & How did they do?
- Let me see a summary of the questions
- Let me see if they got it right or wrong
- Let me navigate to their answer and see the error type

Putting what matters front and center direct from our users

Crafting the performance dashboard experience through feedback sessions with dev math instructors

THE RESULTS

Part 3: Outcome

What were the initial lessons learned?

THE INITIAL E2E VALIDATION PHASE

Results & Lessons

On a truncated timeline, we delivered an adaptive learning experience that personalizes the topics, practice, and instructional scaffolding students receive.

"Overall, it's really nicely displayed. Easy to click around and drill into it the specific chapters to plan lessons."

Allison S., Math instructor

WHAT DID OUR TEAM LEARN?

The bulk of our Rio research has been foundational, where benchmarks don’t apply. Our iterative tests are with limited interaction prototypes and tend to be heavily moderated. Our fast Agile workflow requires flexible and lean testing protocols and emphasizes qualitative insights. Tests for each iteration, then, often look different and ask different questions. A few defined benchmarks:

• Task success (navigation, discovery, and wayfinding) was 100%

These data points are clear successes according to any task success benchmark. We employed the Technology Acceptance Model (TAM) for all user testing. Instructors rate how usable and useful the product is. We chose a shorthand version of this because it’s simple and can still root out strong trends, even with small samples. The TAM score on last end-to-end test was 4.41. Benchmark is the industry-accepted rating that predicts adoption: 3.00.

• Technology Acceptance Model score on final test was 4.41 

"It's very easy to see who needs help and who I would have a conversation with. It's also helpful to see the areas where I could shave off instruction time for high performing chapters."

Chris R., Math instructor

Working toward improvements for the future

Decisive results impact design decisions and our process aim for efficiency

IMMEDIATE USER FEEDBACK

Many instructors told us they spend substantial time trying to identify students who aren’t proficient in fundamental skills like basic arithmetic. Skills “they should have learned in second grade.” They take these deficiencies with them all the way to college. More broadly, instructors want to assess skills that pre-date and are pre-reqs for the current course they’re in. Current diagnostic largely assesses proficiency of the content that will be covered in class.

It'll be important to move from useful and usable results to innovative, workflow-changing insights. We have been successful surfacing and displaying metrics instructors currently rely on and find useful, but we can come up with better and more innovative ways to meet the need behind it. Instead of them sifting through last logins to see who’s been working, we can push alerts that put multiple data points together into an insight that say, “This student has ‘checked out.’ You might want to reach out to them.” Operationally, this might just mean working with PLA to intensify research efforts and apply output.

"The information displayed w/in the tool is helpful and direct. The design is very visually appealing and it's easy to select a chapter and quickly assess where I should focus my time."

Jason B., Math instructor

SOLID START & CLEAR NEXT STEPS

We ran a end-of-the-year retro to find identify possible improvement areas moving into the new year covering:

1. What went well
2. What could be improved
3. Possible improvement solutions for the future
   

There were obvious themes discovered across our retro cards focused on improving overall planning effectiveness and efficiency by transparent communication. With that being said, there were threads of design maturity. 

We all acknowledged that there was stronger collaboration within our working relationship and interactions with dev or product compared to a year ago - or past projects. While also identifying additional tweaks to our design and development practices to ultimately deliver a useful and enjoyable experience.

THE DAY-TO-DAY

Part 4: Behind the scenes

How did we have fun along the way and keep the creativity flowing when solving complex problems?

Craft the day

I started each day with a creative outlet exercise to include our team name in an iconic album cover. I'd then post to Slack with the plan of the day before standup.

It was a great way to get into a creative flow to start the day and engage the team with something new every single morning.

Have ideas? Say hi!

Let's build something impactful together.

hello@zachkulpa.design