The Four-Part Test for Vetting an AI Vendor

“AI is not going to fix a broken process at your company.” That's the first thing Eric deRegt tells practice owners who ask him how to evaluate the wave of AI tools landing in their inbox.

deRegt is the CTO and Co-founder of Indie Health, which he started about two years ago to build technology for independent practices. Before that, he spent three and a half years at a digital therapeutics company where the software products were highly regulated. He has spent his career building software in places where “close enough” was never an acceptable standard.

That background matters, because the market he's describing is loud. Nearly every healthcare vendor has stamped “AI” somewhere on the label, and most of those claims won't survive a serious conversation.

The good news for an independent operator is that you don't need an engineering degree to separate the real systems from the wrappers. You need Eric’s four-part test.

Key Takeaways:

• Be wary of a vendor that brags about its own "proprietary AI models." The vendors worth your time use top general-purpose models and put their effort into tuning the system to your practice.
• A serious vendor can show you who reviews the AI's output, how often, and how fast a mistake gets fixed. If they can't, the system can't improve and neither can your results.
• Tie any tool to one constrained use case and one number you can measure before you sign. For the front desk, that number is the revenue you recover from calls you'd otherwise miss.

AI Won't Fix a Broken Process

The most expensive mistake deRegt sees is treating AI as a silver bullet for deep organizational problems. A tool can take over a workflow. It cannot untangle a broken one. Hand a messy process to a model and you get a faster mess.

The first move isn't to evaluate vendors at all. It's to find the workflows that are repeatable, high-volume, and not the most regulated or sensitive parts of your business — the places where the value is obvious and the failure cases are survivable. Those are where today's models clearly earn their keep.

Once you know which problem you're actually solving, the vendor conversation gets a lot easier.

The Four-Part Test

Part 1. Patient data security is table stakes

Start with the thing that should end the conversation fastest. A vendor handling patient data needs a Business Associate Agreement (BAA) — the contract that legally binds them to protect patient health information under HIPAA. They should offer it on day one without flinching. With many tools for getting HIPAA-ready, hesitation here isn't simply a resourcing problem.

Then, go one layer deeper. Ask whether they hold BAAs with their own providers — the third-party tools they route your data through.

deRegt vendors should know patient data privacy rules extensively and take compliance seriously. If not, you've learned everything you need to know.

Part 2. Separate AI-native from “AI-powered”

Start by taking the model itself off the table. Serious vendors are running their tools on either a large, general-purpose or open-source AI model have basically become commodities.What matters is what a vendor builds around that model, and how well they understand your work.

That's why two opposite-sounding pitches are both warning signs. A “wrapper” is the under-built version — a thin layer of instructions on top of a rented model, with little real engineering underneath, so you pay a premium for something close to what the basic model already does on its own.

“We have our own proprietary AI models” is the over-built version — usually either marketing, or a sign they spent your money rebuilding an engine they could have rented for less.

Worse still are the loudest claims of all: “fully autonomous” and “100% accurate.” “Someone that says they have a fully autonomous system that's a hundred percent accurate. Things like that are just fundamentally unserious,” says deRegt. “Every sophisticated buyer knows these systems make mistakes, and a vendor pretending otherwise is either naive or selling.”

So what do you want? A vendor that takes a top model as a given and pours its effort into the system around it. They've spent real time in your sector. For a private practice, that means understanding what a patient evaluation is and how it flows through the organization. They've tuned the tool to how you actually operate.

That's the real meaning of “AI-native”: a system built around your work, not a generic product with “AI” bolted on.

Part 3. Interrogate the evaluation loop

This is the question most practice owners never think to ask, and it's the one that separates serious vendors from the rest: Who is reviewing what the AI does?

"If I were doing diligence on another vendor, especially on the AI side, I'd want to know who's reviewing the output of the AI," deRegt says. "How often are they doing it? How can I see that as a business owner?"

Push further. When the AI gets something wrong, how do you flag it, and how quickly does the fix turn around?

Serious vendors run simulations and evaluations — structured tests that grade the system against real scenarios, tuned to your practice. Vendors who skip this step and only do light prompt engineering produce less effective tools that improve more slowly, and more importantly, are never tailored to your practice.

At Indie, deRegt's team built internal regression testing so that every change to a voice agent gets tested against every known type of call a practice receives before it ships. The agents run with guardrails and clear escalation points to a human.

Front desk staff and owners are the managers of those agents — they can see what the agent is doing, where it's struggling, and make improvementsweek over week. That's what augmentation looks like in practice: the technology handles volume, and people stay in control.

One more thing to push on is onboarding. Be suspicious of any vendor who promises you'll sign a contract and everything will run smoothly in two weeks. That's not how it works. Ask what onboarding actually looks like and how long it takes to achieve reliable operation. The honest answer is more reassuring than the easy one.

Part 4. Pick one use case and one metric

The fastest way to lose money on AI is to buy everything a vendor claims it can do. Think in phases — what can you realistically accomplish in the first three months, three to six months, a year. Then, look for two kinds of problems worth solving first: the hair-on-fire ones, and the well-defined, repetitive tasks your team hates that quietly waste hours every week.

Next, pick one metric to measure success. Don't let the vendor talk you into measuring ten. The front desk is the cleanest example with a missed call directly impacting revenue. In physical therapy, a new patient might be worth roughly ten sessions across a plan of care, times your revenue per session.

Compare that to the cost of a voice agent answering overflow and after-hours calls, which is usually priced on usage. That gives you a real number to work with.

"Before you sign a contract with a vendor, you want a clear-cut way to tell whether this new tool is making your practice more revenue or more margin," deRegt says.

What Good Looks Like

Put the four parts together and the profile of a serious vendor gets sharp. They sign a BAA on day one and hold agreements with their own providers. They can describe their architecture without hiding behind “proprietary.” They can show you the evaluation loop and tell you the truth about onboarding. And, they'll help you tie the whole thing to one number you can measure.

A vendor who can't do those four things isn't selling you a system. They're selling you a demo. The difference tends to show up about three weeks after the contract is signed. Practices that press on these four areas don’t have to find out the hard way.

Frequently Asked Questions

What's the single biggest red flag when evaluating an AI vendor?

Language that overpromises. “Proprietary AI models,” “fully autonomous,” “100% accurate” — any of these should slow you down. A vendor that brags about building its own AI from scratch is usually telling you it spent money rebuilding an engine it could have rented for less, or it's just marketing.

And anyone who claims their system never makes a mistake either doesn't understand the technology or is hoping you don't. Serious vendors talk about guardrails, escalations, and how they catch and fix errors, not perfection.

We're a four-location PT group with no IT staff. Where do we even start?

Start with compliance — it takes one email to ask a vendor for a BAA and see whether they're serious. Then, pick a single high-volume task where the cost of doing nothing is obvious. For most practices that's overflow and after-hours calls, where every missed call is lost revenue you can actually measure. Solve one thing, track one number, and expand from there. You don't need an IT department to run that play.

How do I judge an engineering team I'll never meet?

Ask about background and process. Have they built software in regulated, complex fields before? deRegt's experience is that engineers who've worked under real compliance and reliability requirements move into healthcare workflows far more smoothly. Teams without that rigor tend to cost you twice what you expected and deliver something you can't use. Then, ask the operational questions: Who reviews the AI's output? How do they test changes before shipping them? And what does onboarding actually look like?

How long until an AI tool actually works in my practice?

Longer than two weeks — and any vendor who promises otherwise is overpromising. Think in phases — first three months, three to six months, and a year or more. A serious partner is upfront about the onboarding process and the time your team will need to reach a reliable state. Slower honesty beats a fast fantasy.