Technical diligence for AI startups

Buying an AI SaaS startup? You’re not just buying traditional software anymore.

Because AI is new, there aren't many best practices on auditing AI companies. Because we've been building with AI here at TechFax, we've got first hand experience seeing where things can go wrong. Here’s what to check before wiring the money.

1. Which AI Models Are Used?

Not all AI is proprietary. Most startups are stitching together OpenAI, Anthropic, or open-source models. You need to know:

- What are they doing to improve outputs? RAG, proprietary data sources, fine-tuning with customer feedback are all great answers. If they're not doing any of these, your moat might be smaller than it seems.

- What models are they using? Are they using the leading most expensive models or have they found a way to get great quality with cheaper models? This will impact costs dramatically. It also lets you know

- Vendor risk? Have they implemented their AI logic in a tightly coupled way or are they using something like the AI SDK or LangChain to abstract away the providers? This reduces switching cost between vendors which is important.

2. Costs & Margins: AI SaaS Bleeds More Than Traditional SaaS

A normal SaaS app might have 80%+ gross margins. AI SaaS? Can be more like 50%, sometimes worse.

- What are the per-query costs? Whatever the primary action is in the tool, figure out the cost for one action. Is this commensurate with the value the user is getting? Braintrust is a great way to monitor AI performance and cost.

- Do we have a gym membership-type problem? Commercial gyms make more money if people don't come to the gym and design around this. Do we have a gym problem here or is usage active? If margins look too good, ensure users are actually using the product.

- Have they implemented any cost savings techniques? Caching, reducing input tokens, or even deleting LLM logic in favor of standard computing are all great ways to reduce costs. If they haven't done this there's lots of upside left in reducing cost.

Look at their cost structure and make sure they’re not upside down.

3. Data: Are They Storing It and Is It Proprietary?

Every AI startup touches data. The real question is: Whose data is it, and is it a liability?

- Data flywheel? Are they collecting and improving their own dataset, or just a UI on top of someone else’s API?

- Do they have access to unique data? Data is a differentiator in AI. If this company has found a unique data set, that's very valuable.

- Are they storing user data properly? This is a question for every startup, but it's worth mentioning here for completeness.

Data is either a moat or a lawsuit waiting to happen. Know which one.

4. Model Evals: Do They Even Know If It Works?

Most AI founders are shipping fast. That’s great—until you realize no one’s doing proper evals. You're going to be changing prompts and plugging in new models. In a non-deterministic system, you need to evaluate its output at each step to ensure quality.

- How do they measure model performance? Precision? Recall? Do they even have a benchmark?

- Do they track degradation? AI models drift. If they don’t have monitoring in place, performance will decline.

- What happens on bad outputs? Is there human-in-the-loop correction, or does the product just break?

If they aren’t rigorously evaluating their model, you should assume it’s not working as well as they claim. Again, lean on Braintrust here.

Handing Off AI SaaS: The Invisible Landmines

Buying AI software isn’t just about closing the deal. You have to keep it running.

1. Swap API Keys

A lot of AI SaaS is just API orchestration. Unlike most SaaS, the OpenAI API key is useful across all domains. You should always be swapping keys after completing the purchase, but with AI it's extra important. When you do that, make sure you have high enough rate limits on your new OpenAI/Anthropic/etc account. Otherwise—incident time.

2. Run Regular Diligence (This Is the Plug for TechFax)

The regular parts of diligence still matter. AI startups have extra complexities, but they also have all the usual SaaS risks—bad code, security holes, technical debt. If you wouldn’t buy a traditional SaaS with shaky foundations, don’t buy an AI one either.

- Code quality and maintainability? If the core logic is brittle, no amount of AI will save it.

- Security risks? AI doesn’t replace basic security best practices.

- Scalability? AI workloads can spike unpredictably—has the team architected for that?

We built TechFax to handle both standard SaaS diligence and AI-specific risks. If you’re buying AI SaaS, you need both. The difference between a great deal and a disaster is knowing what you’re buying.