Why Most AI Integration Projects Fail in the First 90 Days
We've watched and worked on a lot of AI integration projects. The failures all look the same. Here's the pattern and the antidote.
Key takeaways
- Most projects launch without a defined metric the AI is supposed to move.
- Without an eval set, "is it better?" becomes vibes.
- Without a cost ceiling, surprise bills kill the project before adoption.
- Without a rollback plan, the first bad output becomes a crisis.
- The fix is mundane: pick a metric, build an eval set, set a cost ceiling, build a rollback.
The pattern
A team builds an AI feature. It demos beautifully. They ship it. Three weeks later, it's quietly rolled back or kept in a feature flag because nobody can prove it's better than the non-AI version. Six weeks in, the OpenAI bill arrives and is 4× what was expected. By week 12, the team is rebuilding from scratch.
Where it goes wrong, step by step
Week 1: No metric
"We want to add AI", but to do what? Reduce support tickets? Increase conversion? Speed up triage? Without a metric, you can't tell if you're winning.
Week 2: No eval set
You can't trust the demo. The demo always works. You need a set of 100-500 real cases with known-correct answers (or known-good outputs). That's your eval set. Without it, your model swaps and prompt changes are guesses.
Week 4: No cost ceiling
Token costs scale faster than user growth. Without per-user, per-tenant, per-feature caps, one heavy user runs up a five-figure bill.
Week 6: Output safety holes
Edge cases nobody tested. The model says something embarrassing, factually wrong, or worse. Nobody has a rollback.
Week 10: Adoption decay
Users stopped trusting it. Once trust is broken, recovery is brutal.
What works instead
Pick the metric first. Build the eval set second. Set the cost ceiling third. Then build the feature.
This sounds obvious. It's not what most teams do.
What we recommend
For any AI feature, before any code: define the metric, write 50 sample cases with known-good answers, set the monthly budget, design the rollback. Then start prompting.
FAQs
How big should the eval set be? 50 to start; 300+ for high-stakes features.
Can the eval set be generated? Some, but the ground truth must be human-checked.
What metric is best? Whichever business outcome the feature is supposed to move.