MY WORK
Contact

Case Study: Human-in-the-loop ML

Human-Centered Workflows Improves AI/ML Fraud Mitigation

A community college system with 100+ colleges needed to improve adoption of its fraud detection tool—so analysts could work faster, trust risk signals, and strengthen the AI/ML feedback loop that improves early detection over time.

THE CHALLENGE

When Risk Signals Outpace Analyst Workflows

A community college system relies on an internal fraud detection and triage tool to quarantine suspicious applications using ML-based risk signals.

The legacy experience created friction at every step—manual cross-checking across systems, limited search and control over queues, and low confidence in opaque scoring—resulting in inconsistent adoption and a weaker feedback loop for improving detection over time.

Project Objective

Redesign and modernize the fraud triage experience to streamline review, improve trust in signals, and enable iterative enhancement.

Business Impact

Increase adoption and operational throughput so suspicious activity is caught earlier and handled more consistently across a distributed system.

User Value

Give analysts faster, more flexible workflows and clearer context so they can make confident decisions with less cognitive load.

how i led

Role, Team, & Constraints

I led product strategy and experience design in close partnership with platform leadership and engineering, shaping an MVP that delivered near-term workflow wins while laying the groundwork for faster iteration and stronger model operations over time.

1

Product Strategy & Experience Lead

Owned the UX strategy and end-to-end experience direction—from framing the product thesis and “Parity + 1” MVP approach to synthesizing research, validating prototypes, and shaping the phased roadmap.

2

Team Composition

Worked with a cross-functional delivery team spanning product, UX, engineering, and platform stakeholders, plus a cross-college working group of fraud analysts and supervisors to validate workflows and prioritize the highest-value enhancements.

3

Key Constraints

Deliver measurable usability and adoption improvements quickly, minimize backend disruption, and modernize enabling platform capabilities (auth + frontend standards) so the tool could evolve rapidly after MVP. x

WHAT WE DID

Research to Roadmap

Aligned on the primary user and their jobs-to-be-done

We focused the roadmap on the Analyst persona (primary user) and included Supervisors as a secondary persona, clarifying tasks, pain points, and where the current tool broke down in real work.


Designed for a human-in-the-loop system

Improving the UX not only increases adoption and efficiency, it also strengthens the UI→ML feedback loop with cleaner, higher-quality labels—improving model precision over time, boosting systemwide fraud detection, and reinforcing trust in the tool.


Synthesized research into themes that guided the roadmap

We mapped the end-to-end triage workflow, conducted a heuristics review of the current state tool, and explored best-in-class fraud monitoring solutions for inspiration and best practices.

This research and discovery informed six key themes.


Designed a modern triage workflow. Built on a scalable foundation.

We built and iterated high-fidelity prototypes with the working group, focusing on high-leverage workflow improvements:

  • Sortable/filterable application queues to manage review load more efficiently
  • Improved information hierarchy + scannability in application details to reduce cognitive load
  • A grid-driven interaction model (AG Grid) that supports power-user workflows and future enhancement paths

To reduce friction and increase consistency across admin tools, we aligned on enabling technical upgrades alongside UX improvements, including front-end modernization (NextJS) to standardize patterns and speed iteration beyond V1.


Tiered product plan enable near-term wins with a long-term vision

A pragmatic V1 (MVP) anchored in “Parity + 1”—preserve core functionality while introducing targeted enhancements that unlock iterative improvement..

We defined a roadmap that kept momentum after MVP—sequencing V1.x fast follows (including features needing backend changes) and a longer-range path to V2+ capabilities like richer metadata, explainability, and deeper pattern analysis.

V1 (MVP)

Parity + 1 - Preserve existing functionality but design in a more modern, thoughtful way

Prioritize Time-to-Release - Start with out-of-the-box grid functionality and minimal backend changes

V1.x Fast Follows

User requested features that required backend changes

Outstanding discovery themes, including score explanability, fraud history, & accountability

Agile releases to maintain V1 momentum and get enhancements into users’ hands sooner

V2

More complex, high-value user requested features

Richer context data and account levelt metadata

Enable pattern analysis across suspicious applications

V3, V4

  • High value, high complexity features: eg. Rules Engine, ML anomaly detection, etc.
  • User Control, Quality of Life Improvements: eg. Custom Views, Full Screen Mode, Application Notes, User Highlighting
  • Adjust priorities as we learn from previous releases
Success

Outcomes & Impact

01

Faster Triage

Sortable, filterable queues helped analysts focus quickly and reduce time spent hunting for the right data.

02

Stronger Decisions

Richer context and explainability patterns increased confidence and supported more consistent, auditable actions.

03

Smoother Access

Clear MVP specifications through flows, wireframes, and hi-fi prototypes enabled faster development cycles with fewer ambiguities and reduced technical debt.

04

Faster Delivery

NextJS patterns and a product-oriented sprint cadence created a foundation for rapid, iterative improvements across admin tools.

Key Takeaways

Human-centered workflows + explainability are adoption multipliers—and adoption improves model quality when the UI captures better human feedback.

“Parity + 1” balanced quick wins with a scalable path, while a tighter UI→ML feedback loop operationalized CCC’s broader fraud mitigation strategy.

View more case studies:
AI-powered Review Engine | 0→1 MVP | Human-centered AI | Experience Modernization | Human-in-the-loop ML


Let's Create Something Exceptional Together!


Doris Mao Lin | Los Angeles, CA | 310-497-2949 | dorismaolin@icloud.com | linkedin.com/in/dorisl/ | dorismaolin.me