Start with the work, not the tool A product demo shows what a tool can do. This guide starts one step earlier, with a question a vendor has no reason to ask: is there work in your organization that AI would make better, and is it worth what it would cost to find out? Sometimes the honest answer is no, or not yet. That is a legitimate result, and the public frameworks treat it as one. NIST's AI Risk Management Framework says that after its Map function, an organization should know enough "to inform an initial go/no-go decision about whether to design, develop, or deploy an AI system." The UK government's AI Playbook tells teams to be open to "the conclusion that, sometimes, AI is not the best solution for your problem: it may be more easily solved with more established technologies." This guide is for the owner, CEO or managing partner of an organization that holds information it cannot afford to leak and has no AI department. It covers what the controlled studies found about where generative AI helps and where it hurts, how to test your own candidate tasks against that, and when to stop. Law firms and tax practices appear as worked examples. Nothing here is legal or tax advice. Where the studies found real gains A handful of randomized or staggered studies measure what happens to output when people get generative AI. Three are useful here. - Customer support. Brynjolfsson, Li and Raymond studied 5,172 support agents at one company as an AI assistant was rolled out. Issues resolved per hour rose 15% on average, "with substantial heterogeneity across workers." Less experienced and lower-skilled agents improved in both speed and quality. The most experienced and highest-skilled saw "small gains in speed and small declines in quality." Gains were largest on moderately rare problems. - Professional writing. Noy and Zhang gave 453 college-educated professionals mid-level, occupation-specific writing tasks in an online experiment, and gave a random half of them ChatGPT. Average time fell 40% and output quality rose 18%. - Software. Cui and co-authors pooled three randomized trials run by Microsoft, Accenture and a Fortune 100 company, covering 4,867 developers, in which a random subset got a code-completion assistant. Completed tasks rose 26.08% among those using it, with a standard error of 10.3%. The authors say each experiment "is noisy and results vary across experiments." Less experienced developers adopted more and gained more. Read together, they point the same way: gains on drafting, customer-support chat and code completion. In the support and software studies the less experienced gained most; in the writing study the gap between workers narrowed. Each has a limit. The support study covers one company, a Fortune 500 software firm, and one tool. The writing tasks were set online, away from any firm's own files and confidentiality rules. "Completed tasks" counts output, not quality or value. The writing study ran in 2023, and the tools have changed since. Our view is that the direction of these findings will hold up better than the exact numbers. Where AI made capable people worse The same tool can help on one task and hurt on a similar-looking one. Dell'Acqua and co-authors ran a preregistered experiment with 758 knowledge workers, in collaboration with Boston Consulting Group, on realistic consulting tasks. On 18 tasks inside what the model could do, participants using AI completed 12.2% more tasks, worked 25.1% faster on average and produced work of "significantly improved quality." On one complex managerial task chosen to sit outside the model's ability, participants using AI "were 19% less likely to produce correct solutions compared with those without AI." The authors call this a "jagged technology frontier": AI "improves performance for some tasks but worsens it for others, even within the same knowledge workflow and with a seemingly similar level of difficulty." The frontier is hard to see from outside. The authors note the tasks seemed similarly difficult; the results were not similar. METR found a second trap. In early 2025 it randomly assigned 246 real issues from 16 experienced open-source developers, working in large codebases they had contributed to for years, to AI-allowed or AI-disallowed. With AI allowed, developers took 19% longer, with a confidence interval of 2% to 39% longer. Before starting they expected a 24% speed-up, and afterwards they still believed AI had sped them up by 20%. METR is explicit that this does not show AI fails to speed up most developers, and that the study covered software development and nothing else. In February 2026 METR reported on a later study, begun in August 2025, that it calls an "unreliable signal": more developers declined to take part, and 30% to 50% said they held back some tasks, because they did not want to work without AI. Its raw estimates now lean toward a speed-up, with confidence intervals that include no change, and METR calls its data "very weak evidence" for the size of the change. In our reading, the lesson that survives both studies is about measurement. METR still warns that self-reported speed-ups "can be quite unreliable." In Denmark, faster tasks have not yet shown up in pay or hours Task-level gains do not automatically reach the organization. Humlum and Vestergaard linked surveys of chatbot use to Danish administrative records. Workers reported productivity benefits, and employers absorbed AI by reorganizing tasks. Yet the authors estimate "precise null effects on earnings and recorded hours" at worker and workplace level, ruling out effects larger than 2% two years after ChatGPT's launch. That is Denmark, chatbot use, and the first two years. Our view is that it is still a useful warning: someone has to redesign the work to turn saved minutes into something the organization keeps. Figures about how often AI projects fail are weaker than they look. MIT NANDA's 2025 report, read here from a copy hosted outside MIT, reports that 60% of organizations investigated embedded or task-specific tools and 5% reached "successfully implemented", which it defines as tools that "users or executives have remarked as causing a marked and sustained productivity and/or P&L impact." The report calls itself preliminary, rests on a review of over 300 public initiatives, interviews with 52 organizations and survey responses from 153 senior leaders collected at four industry conferences, and says its figures are "directionally accurate based on individual interviews rather than official company reporting." RAND's often-cited "more than 80 percent" is not RAND's own measurement; RAND introduces it with "By some estimates" and cites other sources. Our view is that no precise failure rate is defensible. Evidence on why projects fail is more useful. Why projects fail, according to the people who build them RAND interviewed 65 data scientists and engineers, each with at least five years building AI models, between August and December 2023. From those interviews RAND identified five leading root causes: - stakeholders misunderstand or miscommunicate "what problem needs to be solved using AI"; - the organization "lacks the necessary data"; - the project focuses "more on using the latest and greatest technology than on solving real problems"; - the organization may not have adequate infrastructure "to manage their data and deploy completed AI models"; - the technology "is applied to problems that are too difficult for AI to solve." The study is qualitative, built on what practitioners reported, and covers AI and machine-learning projects in general. Most interviewees were engineers rather than executives, and RAND says the results "may be skewed toward identifying leadership failures." In our reading, the last cause resembles what the consulting experiment found outside the frontier, and three of the five are decided before anyone writes a line of code: the problem, the data and the difficulty of the task. Break the work into tasks before you judge it Judge tasks, not departments. "Use AI in finance" cannot be tested. "Draft the first version of the monthly variance commentary" can. Research on exposure works this way. Eloundou, Manning, Mishkin and Rock broke US occupations into their component tasks and rated each one, using a mix of human raters and a language model. They estimate that with a language model alone, about 15% of all worker tasks could be done significantly faster at the same quality, rising to between 47% and 56% once software built around the model is included. Two cautions come with that. The authors say they "do not make predictions about the development or adoption timeline", and being exposed is not the same as being worth building. In our reading, the gap between 15% and roughly half is the point for a buyer: most of the potential depends on the integration around the model, and that is where the cost sits. We start by listing what people do in a normal week, in their words: what they read, what they write, what they look up, what they check and for whom. Each line is a candidate. Most will drop out at the next step. Seven tests for each candidate task These tests come from the frameworks and studies above. A task that fails one of them is a weak candidate, however good the demo. - A defined problem with a baseline. You can say how long the task takes today, how often it goes wrong, and what better would mean in numbers. NIST asks that "the business value or context of business use has been clearly defined." Without a baseline a pilot cannot succeed or fail; it will drift. - Enough volume. The UK's 2019 guidance asks whether "the task is large scale and repetitive enough that a human would struggle to carry it out." Fixed costs need repetition to pay back. - Inside the frontier. Drafting, customer-support chat and code completion are where the studies above found gains. In the consulting experiment, AI users did worse on a complex task chosen to sit beyond the model. - Data that exists and may be used. The same UK guidance asks whether there is data containing the information you need and whether "it's ethical and safe to use the data." For a professional firm, "may be used" includes client consent and professional rules, which your counsel decides. - A known cost of being wrong. NIST asks organizations to document "potential costs, including non-monetary costs, which result from expected or realized AI errors." The UK Playbook says AI "should not be used on its own in high-risk areas", meaning those that could harm health, safety, fundamental rights or the environment. Where an error is costly, a person reviews the output, and that review time belongs in the estimate. - Checking costs less than doing. If confirming an AI draft takes as long as writing it, the gain is gone. Our view is that this test is hardest to pass in high-stakes work, where review cannot be dropped. - An owner for at least a year. RAND recommends committing a team to one problem for at least a year: "If an AI project is not worth such a long-term commitment, it most likely is not worth committing to at all." Then count the full cost. The UK Playbook warns that, depending on the tool, implementation can carry "considerable compute costs", and says "ongoing investment is needed to maintain these systems." Add integration, permissions, testing, review time, training, and re-testing when the provider changes or retires the model you built on. Two worked examples These are illustrations of the method, not client work. A law firm. Candidate tasks might include a first draft of routine engagement letters, a summary of a long deposition transcript for the partner, finding the firm's own past work product on a point, and analysis of an unsettled legal question. The first two are drafting and summarizing, the nearest of these to the writing tasks where the studies found gains. In each of the three, a lawyer can check the output against the source. The last is novel judgment where a confident wrong answer is the expensive failure. In our view it resembles the task where participants in the consulting study did worse with AI. Retrieval over the firm's own documents also fails the data test until someone settles who may see which matters. Our rule for any such system: nobody reaches anything through AI they couldn't already open. A tax practice. Candidates might include drafting client reminder letters, sorting incoming documents in busy season, and explaining a notice to a client in plain words. Volume is seasonal and often high, which helps. Tax return information carries its own rules: IRC section 7216 makes it a crime for a preparer to knowingly or recklessly disclose information furnished for preparing a return, or use it for anything other than preparing the return, outside its exceptions. So the data test comes first, and your counsel or compliance lead decides it. An answer that goes to a client unchecked fails the cost-of-error test. In both, the useful output is a short ranked list: two or three tasks that pass every test, with a baseline for each. Check whether any of them would be solved as well by a template, a checklist or a better search index. Measure on your own work before anyone relies on it The METR result is the reason to measure instead of asking. In its 2025 study, developers who were slower with AI believed they were faster. A pilot judged on how people feel about it can pass when it should fail. A test worth running is small and specific: - one task that passed the seven tests, done on your own documents; - a baseline taken before the pilot starts: time, error rate, rework; - the same measures during the pilot, with review time counted as part of the task; - a date, set in advance, when you decide to keep it, change it or stop. NIST's framework is voluntary, but its sequence fits: map the context and decide go or no-go, then measure and manage. Stopping at the decision date is a result, not a failure of the pilot. When the answer is not yet Some findings mean the right move is to build nothing for now: - nobody can say what better means in numbers; - the task is rare, so fixed costs do not pay back; - the strongest candidate is novel judgment, where a plausible wrong answer is the main risk; - the data is scattered, poor, or not cleared for this use; - checking the output takes as long as producing it; - no one will own it for a year; - a cheaper fix exists: a template, a checklist, a search index, a process change. The UK Playbook asks teams to focus on use cases where "AI offers significant advantages over existing techniques." If staff already use general AI tools on a task, the first job may be governing that use rather than building something new; the guide on staff using public AI covers that. What a good assessment should hand you Whoever does this work, inside the firm or outside it, you should come away with something you can act on, not a maturity score. Our Diagnostic is how we do it, and we describe it as "the cheapest way to find out whether there is anything here worth doing." It includes interviews, a map of where the work and the knowledge sit, a ranked list of what is worth building, cost and risk, and a written recommendation. Whatever you use, check that the result names specific tasks, gives a baseline and a full cost for each, says what could go wrong, and allows "nothing yet" as an answer. If one or more tasks pass, the next decision is where the system should run and who can see what it reads. Where your AI should run covers that.