Omnisend 复盘编程智能体落地：代码占比超 50% 后为何重新审视度量指标

When you ask a company what their AI investment has given them, many can only offer a subjective answer: tasks seem to get done faster than before. That’s expected, since the big selling point of AI is speed. But “it feels faster” is a business metric in the same way a crystal ball is a forecasting tool.

Rokas Cechanavičius, Engineering Director at Omnisend, says that when you check that sense of speed against real data, such as the volume of code changes, there’s often no meaningful change.

With his team, Rokas didn’t want to judge AI agents’ effectiveness by vibes either. To understand their real impact, he wanted agents in everyday work first, and then to measure what they actually change. That’s how the team landed on a concrete goal for the quarter: 20% of production code changes should be made through AI agents.

The team passed 50%. Rokas’ first move was to question what that number proved.

“We exceeded the goal by a wide margin,” says Rokas. “Still, it’s important to note that this number shows how widespread this way of working has become, not the value it creates for customers or the business.”

You can see that proving real value and business impact is a lot harder than counting adoption. In this story, we share how Rokas and his team navigate that.

Measure the work, not the usage

So what do you do when indicators like the volume of code changes don’t move?

“If these metrics show neither better software quality nor extra value for the customer, that’s a signal to look for a more precise answer,” Rokas says. “Where does the saved time go, and does it turn into a useful result for the organization? If a person finishes their task faster, that doesn’t automatically mean the company delivers what customers need any faster. That’s why we have to measure not AI usage itself, but what work we use it for and what happens once it’s done.”

In March 2026, Omnisend started rewarding employees for meaningful AI adoption. That comes down to three questions: Does AI save time or costs? Does it move business goals forward? Do other people adopt what one person builds? Simply opening an AI tool doesn’t count, and a usage count can’t answer any of those questions. So in engineering, the team looks at what the work produces instead.

Make agent work visible

To measure what agents produce, the team first needed to see that work. So they moved coding agents off individual engineers’ computers and into a shared cloud environment.

This move does three things:

• Rules and work instructions live in one place instead of on each engineer’s computer
• Tasks are easier to rerun under the same conditions they originally ran in
• Changes get another review before they’re applied

It also means every agent run leaves a record: the request, the actions taken, the code changes, and the time spent. Those records make the rest of the measurement possible.

Sort tasks by purpose

Asked where to start, Rokas suggests sorting the tasks given to agents by what they’re for.

“When we clearly define maintenance work, for example updating libraries, fixing documentation errors, or removing specific bugs, we usually get results that are easy to verify,” says Rokas. “Building new features around specific customer needs and product direction usually calls for more individual decisions. Just counting completed tasks, it would be hard to say whether AI freed up time for more valuable work, or simply helped us do more maintenance.”

When you sort the work, that question gets easier to answer. With maintenance and feature work tracked separately, you can see where agents help.

Read the signals together

No single number tells you whether coding agents are working. Rokas suggests tracking several that only make sense together:

• How many proposed code changes get accepted
• How many pass automated checks on the first try
• How often reviewers ask for fixes
• How long approval takes
• Whether defects show up within 30 days of deployment

“This data has to be compared with the agents’ task logs, costs, and work categories,” he says. “Only then does it become clear which tasks get done faster, where extra review is needed, and whether the capacity we’ve freed up is going towards work that matters to customers.”

None of these signals measure how much AI gets used, since usage only shows that people are trying. These numbers show whether it’s working.

Turn the best experiments into shared practice

Knowing what works only helps if good practices can spread, and that’s harder when everyone experiments their own way. Engineers had room to try things with agents, and each wrote instructions for their own needs. The result was overlapping rules, some of which flatly contradicted each other.

“That’s why all the instructions we’ve created are now carefully reviewed and merged,” Rokas explains. “Of course, people need room to experiment. But leaders have to set shared priorities, set a clear quality bar, and build a mechanism for how the best solutions become practice across the whole organization.”

At Omnisend, the instruction an engineer writes to solve their own problem can become the one their whole team works with.

Data over impressions

If an organization can’t say what AI helps it do, or how long it takes to check the results, its AI investment decisions were made on impressions, not data.

“Shared and, most importantly, real data helps leaders understand where task automation pays off, and where the processes themselves need fixing first,” says Rokas. “Of course, there’s no single universal AI ROI metric. It can be different for every organization. What is universal is that evaluating agents has to start with specific work and the result it achieves. Only then will you see how actively a tool is used, and whether it’s changing the quality and speed of work and the team’s ability to create value for the organization.”

For engineers at Omnisend, that’s what working with AI looks like: agents are part of real production work, and nobody takes a good-looking number at face value, including the targets we set ourselves.

The post We beat our AI target by more than double, and then questioned the number appeared first on Omnisend Blog.