LangSmith 推出实时告警功能 Alerts
Catch production failures early with LangSmith Alerts
LangChain 在 LangSmith 中上线告警功能 Alerts,支持对生产环境中的大语言模型应用与 AI 智能体进行实时监控。用户可基于错误数量与错误率、平均延迟及平均反馈分数设置告警规则,支持按模型、工具调用等维度过滤并设定 5 分钟或 15 分钟聚合窗口。告警通知可直接集成至 PagerDuty 或通过自定义 Webhook 推送至 Slack 等渠道。
针对生产环境大语言模型与智能体的依赖故障与质量波动,展示了多维度指标告警的具体配置方式。
Great user experiences start with reliable applications. That’s why catching failures before they reach your users is key. To help you stay ahead, we’ve launched Alerts in LangSmith— making it easier to monitor your LLM apps and agents in real time.
We now support setting alerts based on key metrics like error rate, run latency, and feedback scores.
If you’re already sending production traces to LangSmith, you can set up your first alert today. New to tracing? Get started with tracing in LangSmith.

Why proactive monitoring matters
Monitoring and alerting are critical for any production app— but LLM-powered applications bring unique challenges, which primarily fall into two categories:
Dependence on External Services:
Agentic apps inherently rely on numerous dependencies — you might use one (or many) model providers and have a number of tools such as APIs, web search services, and databases available to your agent. Outages, rate limits, or increased latency from these dependencies can significantly degrade user experience. Proactive monitoring helps you identify these issues quickly.
Quality & Correctness
User experience isn't just about speed; it's also about the quality of the LLM's output. LLMs don't always behave predictably— small changes in prompts, models, or inputs can unexpectedly impact results.
Prompts that perform well in controlled evaluations can also sometimes regress in real-world scenarios due to differences in user interactions. Alerts based on feedback scores (from user input or online evaluations) provide an early warning system for these quality dips.
LangSmith Alerts Overview
LangSmith supports alerting on the following metrics:
- Error Count and Rate
- Average Latency
- Average Feedback Score
For each alert metric, you can leverage a robust set of filters to focus on specific subsets of runs (e.g., filtering by model, tool call or run type).

You can then set an aggregation windows (5 or 15 minutes) and a threshold to tune the sensitivity of alerts.
The last step is integrating the alert into your existing workflows. We support alerts via PagerDuty or setting up a custom webhook (e.g., to send notifications directly to a Slack channel).
And thats it! Check out our docs to learn more and get started today with alerting in LangSmith.
What's Next?
Alerting is a key piece to any observability product. In the future, we will be adding:
- More types of alerts: run count and LLM token usage
- Change alerts that allow you to set a relative value to alert over (e.g. alert when latency spikes 25%)
- Alerts over custom time windows
If you have feedback or feature requests, let us know what you think by getting in touch with us through the LangChain Slack Community. If you’re not part of the Slack community yet, sign up here.
来源:LangChain Blog · langchain.com