AgentsOpsAI is an industry-leading developer platform designed to test, debug, and monitor AI agents. It provides tools for recording LLM prompts, completions, and timestamps, logging events and calls, and linking errors back to their causal events. This platform aims to eliminate the black-box nature of AI agents, offering transparency and insight into their operations through a visual dashboard.

AgentOps AI

Build and monitor reliable AI agents with comprehensive testing and replay analytics.

Langfuse is an open-source LLM engineering platform designed to help developers build, monitor, and optimize large language model applications. It provides tools for observability, metrics tracking, prompt management, and evaluations, enabling teams to iterate on their LLM workflows efficiently. Users can self-host or use a cloud version, making it accessible for both startups and larger enterprises.

Langfuse

LLM engineering platform for observability and metrics


Burr makes it easy to develop applications that make decisions (chatbots, agents, simulations, etc...) from simple python building blocks. Burr helps you ship reliable agents faster than other frameworks with it's built in real-time observability with self-hostable UI and tight developer experience that comes from it. Write code -> tracke it -> see a bug/hallucination -> create a test case for it -> ship reliable agents faster.

Burr works well for any application that uses LLMs, and can integrate with any of your favorite frameworks. Burr includes pluggable persisters (e.g. for memory) to save & load application state, as well as various pluggable extensions.

Burr

Build reliable agents that make decisions. Monitor, trace, persist on your own infrastructure.

Wayfound is your central hub to monitor and improve your AI Agents, so you can scale your AI strategy with confidence.

Wayfound

Supervise, improve, and connect all your AI Agents in one place.

Temperstack is an innovative platform designed to enhance incident management and system reliability. It leverages existing observability stacks to achieve exceptional uptime (>99.99%), automating service catalogs, alert audits, and SLI reporting across multiple observability tools. Temperstack provides a unified command interface for visibility, proactive issue detection, and collaboration across teams, helping organizations maintain focus on core business objectives while ensuring optimal system performance and stability.

Temperstack

Temperstack is a reliability automation  product .

Coval is a simulation and evaluation platform for AI agents, inspired by techniques from the autonomous vehicle industry. It automates testing processes for AI assistants across various modalities, including chat and voice. Coval's platform helps engineers boost test coverage, speed up development, and validate consistent performance of AI agents, addressing the challenges of manual testing and improving reliability.

Coval (YC S24)

A simulation and evaluation platform for AI agents

Building demos of Generative AI applications is deceptively easy; getting them into production (and maintaining their high quality) is not. W&B Weave is here to help developers build and iterate on their AI applications with confidence.

Create rigorous apples-to-apples evaluations to score the behavior of any aspect of your app. Examine and debug failures by easily inspecting inputs and outputs. Deliver high performing AI applications to production.

Weave

Lightweight toolkit for tracking and evaluating LLM applications

Arize AI is an ML observability platform that helps AI engineers and data scientists monitor, troubleshoot, and evaluate LLM models. It enables teams to surface model issues quickly, resolve root causes, and improve overall model performance. The platform supports continuous monitoring and improvement across the entire ML lifecycle, from deployment to production, with features for detecting drift, analyzing performance, and tracing issues back to problematic data

Arize AI

AI observability and LLM evaluation platform for monitoring and improving ML models

LangSmith is a comprehensive developer platform designed to streamline the entire lifecycle of LLM-powered applications. It offers tools for debugging, testing, evaluating, and monitoring LLM applications, helping developers bridge the gap between prototype and production. LangSmith provides visibility into complex LLM workflows, enables collaboration between developers and subject matter experts, and supports continuous improvement of AI systems.

LangSmith

Unified platform for debugging, testing, and monitoring LLM applications

Every cloud-native company faces challenges with software reliability, often hampered by manual, time-consuming engineering tasks during critical incidents. NOFire AI revolutionizes incident management by drastically reducing time-to-resolution through automated root cause analysis. Moving beyond relying on correlation-based approaches, NOFire AI uncovers true causal relationships, even in the most complex SRE environments, enabling teams to go beyond symptoms to address the real issues. Seamlessly connecting to observability platforms, metrics, logs, Kubernetes, databases, and more, NOFire AI integrates with leading LLM providers like OpenAI, Mistral, and LLaMA to deliver a powerful, adaptive solution for modern engineering teams.



NOFireAI

The Incidents Resolution AI for SREs and on-call Engineers battling constant firefighting

Foundry is a comprehensive platform designed to help businesses create, test, and enhance AI agents for various tasks. It offers tools for agent design, performance evaluation, and improvement through auto-prompting, fine-tuning, and steering. The platform aims to simplify the process of identifying weaknesses in AI agents and improving their reliability for real-world applications

FoundryAI (YC F24)

Platform to build, evaluate, and improve AI agents for business automation

Vocera AI is a platform designed to test, evaluate, and improve AI agents. It offers tools for simulating scenarios, replaying real conversations, and conducting evaluations to ensure AI agents perform optimally across various workflows and personalities. The service aims to help businesses identify and fix issues in their AI agents before deployment, enhancing reliability and compliance.

Vocera AI

Monitoring and testing for your Voice AI Agents

Observability AI Agents