Assessment That Works
Ametros Learning was founded in 2016 with a focus on competency-based, experiential learning and assessment for post-secondary education. At the time, that meant building tools to help instructors design realistic scenarios and rubrics that measured what learners could actually do, not just recall information. Over the years, that work grew into partnerships with global publishers, professional certification bodies, and colleges and universities across North America, supporting well over a thousand courses.
As AI became capable enough to genuinely support skills-based assessment, we had a choice: bolt AI onto our existing platform as a shortcut, or rebuild our assessment approach around what we learned makes virtual competency-based evaluation trustworthy. We chose the latter. What follows is some of what that decade of experience taught us, and how it shaped the way we approach AI assessment today.
The problem with grading against general knowledge
Early on, it became clear that the biggest risk in automating assessment wasn't speed or scale; it was drift. A model left to judge answers against its own general sense of "correct" will quietly reward what sounds broadly right rather than what actually reflects a specific program's standards. In competency-based education, that's a serious problem, because the whole premise is measuring against a defined framework, not a vague impression of quality.
Our response was to build an agentic retrieval layer that grounds every assessment in an organization's own approved material; course content, publisher content, professional standards, or internal frameworks. Rather than relying on a model's general judgment, the system checks submissions against what a specific program actually teaches and certifies, and can point back to the exact source behind a given score.
The problem with grading everything as one unit
Another lesson came from watching early automated grading tools produce a single overall score and a paragraph of generic feedback. It looked efficient, but it hid what actually mattered: which specific skill a learner was struggling with. A learner might reason well but communicate poorly, or vice versa, and an aggregate score erases that distinction entirely.
This shaped what became a defining part of our platform: a hierarchical, patent-pending assessment architecture that evaluates work at the sub-skill level first, checking whether a learner demonstrated each specific, narrowly defined capability, and only then rolling that evidence up into a broader competency judgment. It's a more demanding way to build an assessment system, but it's what makes both the score and the feedback behind it explainable and specific enough to actually help someone improve.
The problem with automation replacing judgment instead of supporting it
We also learned that automation is only as trustworthy as the review process built around it. A tool that sounds confident is easy to trust even when it shouldn't be. That is why our platform, Yuta, doesn't treat AI-generated assessment as a finished product.
Specialized agents generate scenarios, rubrics, and knowledge bases from a course brief, separate evaluator agents check that output for completeness and consistency, and a human reviewer approves each stage before it moves forward. Judgment stays in the process; it's just supported by AI rather than replaced by it.
Why this matters beyond education
While Yuta was built and proven in higher education, the underlying architecture isn't specific to any one field. It was designed to assess competency against any defined framework, whether that's a university course outcome, a publisher's certification standard, or a professional body's credentialing requirements. The same questions apply everywhere competency-based assessment does: what is the evaluation grounded in, how granular is it, and how is human judgment kept in the loop?
Nearly a decade of building assessment systems for real institutions and real credentials taught us that trustworthy assessment isn't about the technology sounding impressive. It's about whether a program can stand behind a result months later, explain exactly how it was reached, and show the evidence that led there. That's the standard we've built Ametros around, and it's the one we think the field should hold AI assessment to more broadly as it continues to grow.
About the Author
James White is a Business Development Lead at Ametros Learning, where he works with higher education institutions, publishers, and professional certification bodies to
bring AI powered competency assessment into their programs. He spends much of his time
talking with partners about what actually makes assessment technology trustworthy at scale,
insight that informs how Ametros approaches new partnerships and integrations.