Client work · In production · 2026
Support agent platform
A support agent platform for a B2B software vendor with two product brands. One agent works three channels: helpdesk tickets, where it drafts and a person sends; the vendor's live chat; and a chat widget of its own, embedded on the marketing site and inside the client's apps, where it answers visitors directly. It reads the customer's billing account and the dev board before answering, acts through 20 tools, and learns from what the human agents actually send.
By Suraj Malla · Kathmandu
The problem
A support team for two products, a helpdesk, a billing provider and a dev board. Most tickets were answerable from documentation that existed, but a person had to read the ticket, find the article, check the account, check whether engineering already knew, write the reply, and remember to follow up.
Live chat had become a ticket-intake form: almost no one answered chats directly, and unanswered chats turned into tickets anyway. The vendor's own chat AI, the weakest responder, sat in front of the strongest one.
The brief grew from a ticket assistant into the whole front line, with the team's own judgement kept in the loop where it mattered.
How it works
helpdesk tickets ─┐
vendor live chat ─┼─▶ intake filter ─▶ context ─▶ agent loop, 20 tools
own chat widget ─┘ noise out account · dev board · KB
(vector → rerank → keyword)
tickets: draft posted as a private note, a person sends
chat: answered live, reviewed afterwards
what the human sent ─▶ reconcile ─▶ blind judge ─▶ distil ─▶ candidate
vs what was drafted as-is / which was learnings, approved
edited / better, and by a person before
replaced why they reach the prompt
themes with no answer ─▶ knowledge loop ─▶ draft article ─▶ review ─▶ KB- Three run channels sharing one agent loop, one tool layer, one retrieval path and one analytics store. The first-party widget is an iframe with an HMAC-signed session, SSE streaming, file uploads, and an origin allowlist that drives both CORS and the frame-ancestors policy. Session state lives in the parent page, because Safari partitions iframe storage and a conversation that vanished on reload was the first bug.
- Draft mode for tickets: the agent's reply is posted as a private note. The human copies, edits, and sends as themselves. If the customer writes again first, the draft is marked superseded and rewritten; a draft nobody acts on expires after two weeks.
- 20 tools: read the ticket, search the knowledge base, reply, note, close; look up the billing account, fetch an invoice, apply a discount, cancel a subscription; search the dev board, read an item's comments, create an item; request a trial extension or a marketplace discount; escalate to Slack, notify a partner manager; and on the chat channel, save visitor identity, request a human, open a ticket, add to a ticket.
- A knowledge base store with a lifecycle: draft, in review, published, archived. Only published articles are in the vector index. Every edit bumps a version and keeps the last 20. A daily sync mirrors two WordPress sites into it, with a guard against mass deletion when a site returns too few pages and a guard against mass creation when it returns a flood.
- A Slack interface: the team mentions the agent in the escalation channel to check a ticket, list open tickets, extend a trial, apply a discount, or cancel an account, which needs a second admin to confirm. A direct-message assistant that can only read, by construction.
- A console for the team: a morning brief with emerging issues, a drafts audit trail, a chat inbox with settings and an install guide, the knowledge base with review and history, an evals page for approving learnings, a billing approvals queue, analytics with cost per model and an events log, a system page that says what the agent can do right now, and a test playbook.
Decisions that mattered
5
Feedback is read from what people actually did, not asked for
The team replies from the helpdesk, not from a console. Of 242 drafts in the first month, exactly one was approved or rejected in the console. So the outcome is read back out of the helpdesk instead: the sent reply is compared to the suggested one and recorded as used as-is, edited, or replaced. Nobody has to rate anything.
Adoption is not quality. A human writing something different is not evidence the draft was worse. Quality is decided separately by a blind judge that sees both replies in alternating order, because a judge always shown the same position develops a position bias. It is only asked why the two diverged when the agent lost, and that question is deliberately not blind, so the reason can name the replies directly.
Judged evaluations go through a distiller that proposes candidate learnings. A person approves them. Only approved learnings are injected into the system prompt.
Grounded, and stricter where nobody is watching
On every channel, product specifics must come from a retrieved article, and the reply carries its URL. If no article covers the issue, the agent asks diagnostic questions or escalates. It never invents steps.
On the first-party chat channel nothing is reviewed before it is sent, so the rule is absolute there: no article containing the answer, no answer. Review happens after the fact in the chat inbox.
The chat channel also has no reply tool. The model's final text is the message. Elsewhere a reply must be an explicit tool call, and one chat-tuned model repeatedly researched an answer, logged a note claiming it had sent it, and sent nothing. Two rounds of prompt hardening didn't fix it. Removing the tool removed the failure.
Actions that cost money are requests, not actions
Trial extensions and discounts send a request that a person approves, in Slack or in the billing queue, before anything runs against the billing provider. Cancelling an account from Slack requires a second admin to confirm. Cancelling a subscription in a retention flow requires the customer to say so explicitly; the agent never cancels on silence.
Ticket and account identifiers come from the assembled context, never from model output. Dry-run is the default. A master stub switch returns canned data from every external system, and each integration has its own write gate, so the systems went live one at a time.
The system page in the console shows, for every capability, a state, a plain sentence about what it means for a customer, and the setting that changes it. Five LIVE badges had turned out to be the wrong question: a badge can't tell reading a board from writing to it.
Spikes are arithmetic, not opinion
Each ticket carries a short topic label written by the light model. What counts as an emerging issue is pure counting: at least 3 tickets in 24 hours and at least 3 times the daily average of the previous 14 days. An ordinary busy day doesn't cry wolf, and the ranking is explainable to whoever reads the brief.
Each spike shows whether the knowledge base already answers it. In the KB means customers can't find an answer that exists. No article means one needs writing. The week's unresolved tickets are grouped by theme so that one article closes a group rather than a ticket.
One agent, two brands
A brand profile bundles the identity the agent introduces itself with and the slice of the knowledge base it may retrieve from. The two profiles are deliberately asymmetric: one brand's surface gets a hard filter so a question about the other product gets a redirect, not an answer; the portfolio surface gets no filter. Everything else, the loop, the tools, the learning, the review, is shared.
What is and isn’t shown
The client and its products aren't named. The 242-drafts figure is an adoption fact from the code, not a performance claim. Ticket volumes, deflection rates and the agent-versus-human benchmark results belong to the client and aren't published.
Next step
Have a system like this in mind?
Describe it: what it is, who does it, how often, and what goes wrong when it's late. I reply within one working day with a scoping call or a reason it isn't worth automating.