Part 1 (of 2): From Demos to Durable Systems

The rise of Generative AI has sparked a wave of experimentation. Within months, teams have gone from toy demos to prototypes capable of natural language reasoning, retrieval, and automation. But when it comes to moving these agents into production? That’s where many efforts stall.

Just as the early days of cloud and containerization exposed new operational challenges, GenAI is revealing that old patterns must be rethought. However, this shift isn’t just about technology. It touches organizational structure, DevOps culture, evaluation practices, and tooling.

I am going to try and explore my point of view as to what stays the same, what changes, and what it truly takes to run GenAI applications at scale.

The Traditional Software Playbook: Deterministic and Engineered

Running traditional software in production is a well-understood game. Deterministic inputs produce expected outputs, or as my CS professor in university said: ”When you (SW Dev) get pissed at the machine, remember, it does exactly what you told it to do, it doesn’t know anything else.” Testing is systematic. CI/CD pipelines, QA gates, alerting, versioning, rollback – all evolved to serve a world where behavior could be predicted and engineered.

These foundations matter just as much today. But GenAI introduces uncertainty, variability, and non-determinism that strain these foundations.

GenAI Agents: A Different Class of Applications

LLM-powered agents are:

• Non-deterministic: Responses vary across identical inputs.
• Prompt-driven: Logic is often embedded in plain language, not code.
• Data-dependent: Outputs shift with user input, training data, and external context.
• Behavioral: Success is judged by tone, reasoning, and task success, not just functional correctness.

This makes observability, testing, and iteration profoundly different.

What Stays the Same

Despite these differences, core software principles still hold:

• CI/CD: You still need structured release processes, now extended to include prompts and evals.
• Testing: QA remains critical, it’s actually back at its glorious day, but must account for fuzzy logic and probabilistic reasoning.
• Monitoring: Latency, API failures, and cost matter – but so do hallucination rates, task success, and prompt drift, perhaps even further important.
• Versioning: Tracking changes to prompts, models, and context windows is vital for reproducibility.

What’s New (and Non-Negotiable)

To run GenAI in production, you need to manage:

• Prompt management: Prompts are an A-class artifact in your app dev process and as such they must be versioned, tested, and updated as product logic.
• Evaluation frameworks: Automatic and human-in-the-loop evals for truthfulness, tone, reasoning quality.
• Data & feedback loops: Capturing logs, labeling outcomes, and feeding insights into improvements.
• Model-aware pipelines: Treat models like dependencies, capable of behavior shifts even with no code changes.
• Semantic QA: Evaluate similarity, intent alignment, and task effectiveness, not just string match correctness.

Organizational Shifts: New Roles, New Culture

About 15 years ago, while I was still at VMware, I started talking about the new emerging operating model in light of cloud computing – DevOps. Most people told me I was BSing, “there’s Dev and theres Ops”, 3-4 years later it was clear, any company running on cloud, built DevOps teams.

Well, just as DevOps brought software engineers and operators together, GenAI demands cross-functional teams:

• Prompt Engineers & Evaluators work closely with PMs and developers.
• AI-QA Analysts test behavioral performance, not just edge cases.
• Human Feedback Loops become central to iteration.

Tooling must support this convergence, allowing teams to debug, test, and improve model-driven logic just like they do with code.

Why Tooling Must Evolve

You can’t run GenAI in production with just Git, Jenkins, and unit tests. Purpose-built infrastructure is essential to:

• Track prompt and model changes
• Run semantic regression tests
• Collect behavioral metrics over time
• Automate evaluations and CI for natural language behavior

GenAI systems require a new layer of tooling – one that bridges experimentation with reliability, enabling teams to iterate on language-driven logic the way they do with code. These platforms must support versioning, evaluation, deployment, and monitoring of agents in environments that are fundamentally probabilistic and context-sensitive.

Conclusion: Productionizing Intelligence

Shipping GenAI agents to production isn’t just an ML problem or an ops challenge. It’s a whole new discipline. One that blends the rigor of software engineering with the ambiguity of human language.

Teams that embrace structured evaluations, behavioral QA, and model-aware DevOps will build trustable, scalable AI systems. But, you know, it also doesn’t end there, because technology and is evolving so fast in this era, that you need to set a framework that allows you not just to build something, but have an ongoing assessment mindset, those who can constantly test new approaches and technologies, are the ones who will come out victorious.

In Part 2, I’ll walk through what it takes to run GenAI agents reliably in production. Stay tuned.