AI Isn’t Software. Stop Evaluating It Like It Is.

Most multi-location operators have bought software before — a scheduling system, an EMR, a CRM. The evaluation is often the same: Does it have the features you need? Does it integrate with what you already run? What does it cost?

Each has a clear, fixed answer you can check off a list. While you can buy software this way, it is the wrong way to buy AI.

Eddie Czech, Co-founder and CEO of Indie Health, explains why this approach doesn’t work, and what to do instead.

It starts at the foundation. "I'd be shocked if more than 20% of practice owners actually understand the distinction between how AI technology is built and how traditional software is built,” Eddie says. And that’s a problem because it isn’t a technical footnote, it shapes everything downstream, including how you evaluate a tool before you buy. "It's very different from building traditional software, and I think people still make the association that this is another software vendor, but it's just dramatically different."

All it takes is a simple mindset shift to overcome the urge to compare AI to more familiar software systems. Eddie offers four key areas to focus on.

Key Takeaways:

• AI is probabilistic, not deterministic. Software gives the same output every time; AI gives a range. So you're evaluating how it behaves across many situations and the risk you can tolerate — not checking a feature off a list.
• Evaluate the team before the product. You can't fully judge a probabilistic system from a demo, so the people who built it — their track record, industry experience, and how they test — are your real signal of quality.
• Perfect is the wrong bar. Think of AI implementation in terms of workflows and judge its performance against human capabilities, not perfection.

Software Is Binary. AI Is Not.

Traditional software is deterministic. A person wrote explicit rules, and the program follows them exactly. Feed a deterministic system the same inputs and you get the same outputs on the first call and on the ten-thousandth; if it worked in the demo, it will generally work that way in practice, because it isn't deciding anything, it's following instructions someone typed.

Eddie explains the difference: "Software historically was very binary … It's very, very different with artificial intelligence. There's a spectrum of outcomes that can happen because it's all probabilistic." Instead of running on hand-written rules, AI is trained on enormous volumes of examples and, in the moment, predicts the response most likely to fit. It's playing the odds on the best answer, not executing a fixed instruction.

The upside: AI can handle messy, unscripted situations no one could ever have written a rule for — a patient who calls with a question your script never anticipated. The trade-off is the same input won't always produce an identical output. A voice agent can take the same call twice and word things differently or make a different judgment at the edge of what it knows. That isn't a defect; it's the nature — and the power — of the technology.

And it changes your evaluation. With software, a demo is practically a guarantee. With AI, a single clean demo only tells you how the system behaved once, not how it handles thousands of real calls. So "working" stops being a box you check and becomes a question of range: How does the system behave across the full spread of situations it will actually face? That range isn’t fixed either. A serious team narrows it on purpose, with guardrails and controls, so probabilistic doesn’t mean unpredictable.

The New Lens for Evaluating AI

Vet the Team, Not Just the Product

The evaluation starts with the team, not the product. If you can't fully judge a probabilistic system by watching a demo, the people who built it become your best read on quality. A probabilistic product keeps making judgment calls in the field and keeps getting updated, so you're really choosing a partner you'll depend on long after signing.

This is the step Eddie says buyers skip. "I don't think enough people spend time evaluating the underlying team behind the product," he says. “Most default to the pitch and never get to the questions that matter.”

So, what should you look for? Eddie says start with regulated-industry experience. A team that has built under HIPAA, PHI rules, and payer requirements designs for those constraints by instinct. Eddie has seen “vendors outsource a lot of their engineering to offshore teams that don't understand healthcare in the U.S." When that's the setup, he says, nine out of ten times those relationships run into trouble.

Next, domain-matched expertise. Because these products are novel, quality depends on whether the developers have made this thing before. Eddie advises being specific: "If it's a voice agent, have they spent time in voice AI?" The people worth trusting have shipped this kind of system, or something adjacent and understand it deeply. He has seen a wave of new entrants "claiming a lot of things without understanding the underlying components of how these products are built."

That experience gap is easier to hide than ever, because AI coding tools used to build these products can churn out code fast. “Output does not mean quality whatsoever," he cautions. A slick-looking product can sit on shaky ground. What separates the two is whether the team can actually read and judge the code underneath, not just generate it.

Understand the Testing Regimes

A team you trust gets you halfway. The other piece is proof the product holds up — and this is where AI departs most sharply from software. You can't test AI using the same approach you'd test software: run it, confirm it matches the spec, ship it.

"Any good vendor is running what are called evaluations," Eddie explains. “Structured simulations that stress-test how a system behaves before it ever reaches a patient.” At Indie, for example, evaluations happen at scale: "We have the ability to run thousands of simulations, which is basically we're taking two voice agents and having them have a conversation with one another."

For a probabilistic system, evaluations are quality control. Running the system against a huge spread of scenarios — the confused caller, the angry caller, the one with a tangled insurance question — is the only way to see how it behaves at the edges.

Ask any vendor to walk you through how they run evaluations, and to show you the results for a workflow like yours. Once you know how a system holds up in the hard cases, you can judge how much you trust it — and decide how broadly, across which situations and which workflows, you're comfortable letting it run.

Stop Thinking in Features. Start Thinking in Workflows.

You've found a team you trust. You’ve confirmed they run thorough evaluations. Now, there's still the question of what you point the tech at. Answering that question means dropping the deepest software habit of all — shopping for features.

Evaluating software is a checklist. Evaluating AI is a workflow map.

"If you go to a software vendor, you ask, 'Do you have a CRM that I can organize all my patient information in?' That's a very binary question. But with this technology, it's more like, 'Do you guys do referral management?' And then there's a series of things that must happen to manage referral appropriately," explains Eddie.

A single referral, in Eddie's example, isn't one feature — it's a chain. An agent reads the inbound fax and pulls out the referring physician, the patient, and the treatment. A texting agent reaches out to schedule the first consultation. A scheduling agent books it. A voice agent may handle intake. Each is a different piece of technology doing a different job inside one workflow.

"You should think about this technology in terms of workflows and not necessarily features," he says. Miss that, and you'll buy a "feature" that solves one step of a five-step process — then wonder why you aren’t seeing the transformation you envisioned.

Perfect Is the Wrong Bar

The expectation of perfect performance is the last piece of the software mindset to drop. Eddie often gets asked if “Agents are 100% accurate?” But he advises perfection is not the right goal.

The more useful measure is human performance. “Is your front desk staff 100% accurate?” As Eddie puts it: "You have to compare it to how well a human would do this, and how quickly would a human do this?” A front desk fielding its fortieth call at 8 PM isn't running at 100% either. Nobody expects staff to be flawless. You expect them to be good, and you build around the gaps.

The right question is whether the technology is faster, cheaper, and accurate enough for the risk level of a specific workflow. A 2% miss rate on something low-risk like after-hours scheduling is a rounding error — and if it's faster and cheaper than a person, it's a win. The same miss rate on something high-stakes is a different conversation.

The urge to insist on 100% before you'll deploy a tool has repercussion throughout the practice. Deploying AI is not about replacing staff. It's about handing the repetitive, low-judgment work to something faster and cheaper so your best people can do what grows the practice. "Why don't you have your best folks spending more time on physician outreach and building better referral networks?" Eddie posits. “Instead of triaging the fax that just landed in the inbox, put them on renegotiating payer rates.”

Start Now

Step back and the shift is smaller than it sounds. As Eddie puts it: "It's a new lens that you have to evaluate these technologies through, which is definitely a shift for a lot of people." Judge the team before the product. Ask how they test it. Buy the workflow, not the feature. Measure it against your staff, not perfection. That's the whole paradigm.

And you don't need certainty about where any of this is going to start using it. Eddie says don't dismiss it for today's rough edges, but don't buy the hype about what tomorrow could bring either. His advice on AI is just to begin: "You should get started using AI in some capacity today." The mystery clears fast once you do. Up close, these tools aren't an overwhelming unknown — they're just another thing you can learn to run.

Frequently Asked Questions

What's the single biggest mistake practice owners make when evaluating AI tools?

Treating them like software. Traditional software is often deterministic (same input, same output), so you evaluate it with a features checklist. AI is probabilistic, meaning the same input can produce a range of outputs. That changes the whole evaluation. You're assessing how a workflow behaves across many situations and how much risk you can tolerate when it hits an edge case, not whether a box is checked.

Why does the team behind the product matter so much more than with regular software?

Because AI products are novel and non-deterministic, quality depends heavily on engineering judgment that isn't visible in a demo. And a probabilistic system isn't a one-time purchase — it keeps making judgment calls in the field and keeps getting updated, so you're really entering a partnership you'll rely on long after signing. That makes the team as important as the product. Eddie's advice is to evaluate whether they've built high-quality products before, whether they have experience in a regulated industry, and whether they're keeping the hard engineering in-house rather than outsourcing it to teams that don't understand U.S. healthcare.

What is an evaluation and why should I ask a vendor about it?

It's how a serious vendor proves a system works before it goes live. Instead of confirming the product does one thing correctly, the vendor runs it through a huge range of simulated scenarios to see how it behaves across all of them. Evaluation protocols are your clearest read on reliability: a vendor who can walk you through their regime can show you how the system holds up in the hard cases, and the ones who can't probably don’t understand their own product.

How accurate does an AI workflow need to be?

Not 100% — that's the wrong bar. The benchmark is the person doing the job today: If the AI is faster, cheaper, and at least as accurate as your staff, it's a win, even with the occasional miss. How high the bar sits depends on the stakes. A small error rate on low-risk work like after-hours scheduling barely matters, while a high-risk workflow demands more accuracy and a human kept in the loop.