The model impressed you in the demo. Will it work in your company?

Sources checked on 12 October 2026.

On the screen in a meeting to choose an AI assistant for the procurement team, the test starts with a familiar case. The vendor uploads three quotes, and within minutes a tidy comparison and a clear recommendation appear. Everyone in the room knows how long this comparison usually takes, so the decision looks easy. Once the trial begins, one quote arrives in a different currency, another leaves out a core service, and a third changes its delivery date in a separate message. The comparison still looks good, but the employee rebuilds it to check what went into the calculation.

This is a hypothetical example, but it puts a specific question to the buyer: which part of the work did the demo prove the system can do? Finding the numbers, understanding the exceptions, or delivering a comparison fit for a decision? Each answer needs a different test. And if the definition of success stays vague, the evaluation may end up measuring what the system presents well, while the team bears the cost of whatever is left undone.

Read the name of the metric before the number

In its 12 October 2026 announcement of GLM-Legal, Z.ai reported a 19.2% all-pass rate on the Harvey LAB benchmark, 12.5 percentage points above the base model. This is a result published by the vendor, not an independent check of performance inside your company. The announcement also describes improving the model and the agent's runtime environment together, so the whole improvement cannot be credited to the model alone. [1]

Harvey defines an all-pass as succeeding on every criterion used to grade the task. The figure here is not "answer accuracy", and it is not a general verdict on every possible use. A system can succeed on many parts and still fail to deliver the whole task. This distinction matters to a manager because it returns the discussion to what the team will actually receive, and to how much work will still be left for them afterwards. [2]

Choose the cases that reveal the real work

Before the trial, ask the process owner to pick a small, varied set of past tasks, prepared for authorised use. Include routine cases, missing information, conflicting documents, and a case with no reliable answer. For each one, write a description of the acceptable output, and have someone who knows the work review it. This is a proposed acceptance list, not a general standard that suits every company.

In the procurement example, the task may require converting all quotes to the same currency, showing excluded services, linking every value to its source, and flagging a delivery date that has not been settled. Decide in advance what prevents the comparison from being accepted. Omitting a material cost may stop its use, while a confusing heading can be fixed quickly. This separation stops polish in the presentation from hiding an error that changes the decision.

A full pass starts with reading every criterion

One decisive criterion that is not met can prevent the whole task from being accepted. Hypothetical data to illustrate the method of evaluation, not results for any product or vendor. This is a single case only.

The grid shows a hypothetical model for evaluating two solutions across different requirements; its cells are illustrative examples of our own, not results for GLM-Legal or any other product. Read each row as a condition for the work. A large number of good cells does not make up for a failed decisive criterion. And the team can agree on those criteria before vendor names enter the discussion.

Also keep back cases the vendor did not see during setup. Give them a fair chance to connect and tune the system, then test it on these new cases, with the same inputs, time and permissions it will have in its intended use. And record every piece of help the team gives during the run. If a good result needed an expert to step in, that is part of how the solution works and what it costs, and it must show up in the evaluation.

Make the weak points visible

One solution may excel at extracting information and weaken when handling an incomplete file. Another may write a less fluent explanation but pinpoint where it is unsure and ask for clear intervention. Combining these differences into one number makes comparison easier, but it may hide the reason one solution suits a particular task. That is why an evaluation deserves more than a final ranking of vendors.

The radar chart shows five proposed dimensions: accuracy of the information, completeness of the task, whether sources can be traced, handling of exceptions, and delivery of the output to the next team. The scores are hypothetical, for illustration. Use the chart to open a discussion about the spread, then go back to the cases that produced the scores. The area of the polygon is not a purchasing decision, and the dimensions do not carry the same weight in every process.

Also define what each level means before scoring. For source tracing, for example, having a link is different from a reference that reaches the passage that actually supports the claim. For delivering outputs, creating a file is different from creating a file a colleague can use without repairing the calculations. And when reviewers disagree, keep the reason for the disagreement; the definition of the task itself may need revisiting.

Test the behaviour when completion is not possible, too. What happens if the system cannot find the necessary document, two sources conflict, or access has expired? Asking for the missing information may be the right action. Record that outcome as a proper escalation under the test design, and do not mix it with completing the task. That way you measure the quality of execution and the quality of stopping, each in its place.

Count what reaches the end of the process

After evaluating quality, compare the trial with the current way of doing the work. How long did it take to reach an acceptable output, including review and correction? Who stepped in, and what did they fix? How many times was the task sent back to the start? Record the cost of running and reviewing together. A fast draft may be useful, but its value depends on the work it saves once you count what the team needs in order to approve it.

Then turn the result into a decision with a defined scope. The solution may suit extracting quote data, while the final comparison stays with the employee. Or it may succeed on a recurring category of requests and need a different route for the rest. That definition lets the trial expand gradually, and stops the success of an easy case from becoming a promise that covers everything the department does.

Document the model version, the run settings, the connected tools, and the type of files used. When one of these changes, rerun the most important cases before expanding use. Your decision concerns a system working under known conditions, and a drop in results will be hard to explain if those conditions changed and nobody kept a record of them.

The best question in a procurement meeting: show us a task from our own work, completed to the conditions we agreed.

Ask for the meeting to end with a short document signed by the process owner: the target task, its acceptance criteria, the cases that will be tested, and who is responsible for approving the results. Attach one output that succeeded and one that stumbled, with an explanation of the difference. Management then has a basis for funding a wider trial, narrowing it, or stopping it. And the capability you buy can be described in the language of the work: a specific output reaching a specific person, with measurable effort, at a time you can rely on.

Sources

• [1] Z.ai - GLM-Legal: Advancing Agentic Legal Intelligence. Vendor announcement, 12 October 2026. The 19.2% figure is the all-pass rate on Harvey LAB; the improvement is 12.5 percentage points. The announcement describes improving the model and the agent's runtime environment together. The figures are not an independent evaluation.

• [2] Harvey - Open-Sourcing Harvey's Long Horizon Legal Agent Benchmark. Explanation published on 6 May 2026. An all-pass requires every grading criterion for the task to be met.

The legal context is used here to illustrate measurement when buying enterprise systems; the examples in this article and the proposed evaluation tools are our own.