Four legal research failures small firms should ask AI vendors about

Legora says it has catalogued more than 50 ways AI can fail at legal research. Four examples deserve particular attention: missing an amendment, treating a dissent as the holding, giving a non-binding agency decision the weight of a Supreme Court judgment, and relying on a passage that a later case has made obsolete. These examples were reported by Artificial Lawyer.

The problem is not limited to invented authorities. A source can be genuine and still be used incorrectly because its status, weight or current validity has been misunderstood.

When assessing a legal research tool, ask the vendor:

1. How does the system rank different types of authority?
2. How does it check whether legislation has been amended or a case has been affected by a later decision?
3. How does it distinguish a majority holding from a dissent or other non-binding material?
4. Can the user inspect the source and the relevant passage before relying on the answer?
5. What happens when the system cannot resolve one of those questions confidently?

The answers should describe a process that a firm can examine and test. General assurances about accuracy are not a substitute for source access, clear limits and appropriate human review.

The Solicitors Regulation Authority's warning notice on misuse of AI says that using AI does not remove professional responsibility for the quality and accuracy of work. It also says that named case authorities used in court submissions should be genuine, relevant and supported by a verifiable citation.

Margo should be judged against the same practical standard. Any contract-review feature should define relevant failure modes, preserve source traceability, support repeatable testing and keep human review in the decision process. Those are requirements to verify, not claims that errors cannot occur.

Hamza Suleman spent three and a half years inside a large international firm.