Here is an uncomfortable way to think about the feature I had just built. A natural-language interface over your operational and financial data is a polite query tool. Anyone with a login can point it at anything. The same fluency that makes it useful also makes it dangerous. The bot tries to be helpful. Helpful and authorized are not the same thing.
Part 3 was about teaching the bot good judgment through prompts. This post is about not relying on that judgment for anything that matters. Prompts shape behavior. They do not enforce it. You can talk a model out of a prompt. You cannot talk it out of a C# authorization filter that refuses to run the query.
That distinction is the spine of everything below. Guidance goes in the prompt. Enforcement goes in the code.
Authorization is mapped, not improvised
BargeOps had a real authorization model long before the chatbot existed. Role-based filters are applied as attributes on controllers and actions. Dispatch, accounting, marketing, logistics, and more. A salesperson does not see AP internals. Billing roles gate financial data. This model was battle-tested.
The mistake would be to let the chatbot bypass all of it. Requests now arrive as English instead of HTTP routes, but that changes nothing about who is allowed to see what. So I mapped every query type the bot can run to the authorization filter that governs it. This happens in a dedicated AuthorizationValidator:
"calculate_trip_margin" => "FinancialAuthorizationFilter",
"get_outstanding_balance..." => "FinancialAuthorizationFilter",
"get_idle_barges" => "AllUserAuthorizationFilter",
"get_trips_pending_approval" => "ApproverAuthorizationFilter",
// ...
When the model decides to call calculate_trip_margin, the executor does not just run it. First it checks whether this user is allowed to run that class of query. It uses the same filter the equivalent screen would enforce. The AI never becomes a privilege-escalation path. It is wired into the exact authorization story the rest of the app already uses. The chat interface is new. The permission boundary is not.
This is the enterprise .NET dividend again. I did not design a new authorization model for the chatbot. I mapped the chatbot onto the one that already existed and was already trusted.
The guard that runs before the model
Some requests should never reach the language model at all. This is not "let the model handle it carefully." The model should never see them.
Write intent is the clearest example. ChatBot is read-only by design. It answers questions. It does not change data. But users will type "void trip 1234" or "update the rate on contract ABC" or "delete those duplicate bills." If you route that to the model and hope the prompt holds, one clever phrasing can cause trouble. You also pay for a model round-trip just to produce a refusal you could produce for free.
So the strongest guards fire before any tool or LLM call. The git history shows where this hardened:
fix(chatbot): refuse write-intent requests before any tool/LLM call
fix(chatbot): harden write-intent guard priority ordering
fix(chatbot): refuse too-broad requests before any tool/LLM call
A request to mutate data is caught at the door. The bot redirects the user to the actual screen where that operation lives, with its real permissions and real audit trail. The model is never consulted, because there is no judgment call to make. Read-only means read-only. The cheapest and safest place to enforce that is before the expensive, probabilistic part of the pipeline.
The principle is simple. The safest guard is the one that runs earliest. If you can refuse a request deterministically, refuse it deterministically. Do this before you hand control to a system whose output you cannot fully predict.
The leaks you only find by looking
I want to be honest about the messy middle. The git history does not lie, and a tidy version of this story would not help you.
The demo bot leaked. Not maliciously. It just answered too eagerly. The problems only became visible under real data and real roles. All of them surfaced in our own QA cycle, before the feature reached users. That is what the cycle is for. The commits that fixed this are some of the most important in the project:
fix(chatbot): role-aware denial + close financial-data leaks
fix(chatbot): scope outstanding balance query to named customer
fix(chatbot): filter status-scoped chat queries to matching records only
feat(chatbot): customer scoping for trips pending approval
fix(chatbot): scope trips-pending-approval to last 90 days
Read those as a group and you can see the shape of the problem. In each case the bot returned more than it should have. An outstanding-balance question was not scoped to the customer the user named, so it pulled balances broadly. A status-filtered query returned records outside the requested status. An approval queue was not bounded to a sensible time window, so it dragged back history nobody asked for.
None of these are exotic AI failures. They are the same over-fetching and under-filtering bugs you find in any data layer. But they are worse in a chatbot, for two reasons. First, the natural-language interface hides the query. A user cannot see that a filter is missing, the way they would notice a wrong filter on a grid. Second, the model's confidence launders the result. A wrong answer arrives in a fluent, authoritative sentence, with none of the visual cues that make a suspicious grid row catch your eye.
The fix was the same every time. Push the scoping down into the deterministic layer. Bind the query to the named customer. Enforce the status filter on the records actually returned. Bound the time window. Do not trust the model to scope correctly. Verify it in code.
QA against a chatbot is its own discipline
Testing this was not like testing a normal feature. I underestimated it.
You cannot enumerate the inputs. There are infinite phrasings. So I stopped testing inputs and started testing behavioral categories, the same categories the failure templates from Part 3 define. Does a write request get refused, no matter how it is phrased? Does a calculation request get refused and redirected? Does a financial query respect role boundaries? Does an explicit identifier correctly drop the prior conversation scope? Each of those is a property. I can probe it with many phrasings and one expected behavior.
The history shows how iterative and humbling this was. There is a sequence of QA-failure-template fix commits, then a revert, then a re-apply:
Merged PR: QA failure template fixes
Revert "Merged PR: QA failure template fixes"
Merged PR: revert of the QA failure template fixes
fix(chatbot): unify calculation refusal across all paths
fix(chatbot): allow summarize, refuse calculate
That revert is the interesting one. A batch of fixes shipped. Something regressed. The whole batch came out. Then the fixes went back in more carefully, one path at a time. "Unify calculation refusal across all paths" is the tell. The calculation guard was correct on one code path and missing on another. So the same prohibited request behaved differently depending on how it arrived. That kind of inconsistency is invisible in a demo and obvious in QA. It only got fixed because someone tried the same bad input through several doors.
I leaned on agentic workflows and SpecKit to drive this. Each fix had a spec, a plan, and tasks. That kept the behavior changes documented and reviewable instead of improvised. But the discipline is the point, not the tooling. You verify a chatbot by deciding what it must never do. Then you attack those properties from as many angles as you can invent.
Audit everything, because someone will ask
Every query the bot runs is logged. The user, the session, the query text, the query type, and the timing. A ChatQueryLogService exists for one job: to make the bot's activity reviewable. The bot can even answer questions about its own audit log for authorized users.
This is not optional in an enterprise context, and it is not just for incident response. Someone will ask, "Can this thing see payroll?" or "What has it been used for?" The answer has to be a query, not a shrug. A feature that touches financial data without an audit trail is not a feature you can defend in a review. Build the logging in from the start. Retrofitting it later is miserable and incomplete.
The mindset shift
The main thing I want you to take from this post is a reframing. When you build a chatbot over real business data, your job is not to make it answer well. That is the easy part. Your job is to bound what it can do, scope what it returns, refuse what it should not touch, and log all of it. And you put every one of those guarantees in deterministic code, using the authorization and validation infrastructure your enterprise app already has.
The prompt makes it polite. The code makes it safe. Do not confuse the two.
In the final post, I will cover what it took to make this fast and durable enough for daily use. Caching, performance monitoring, conversation memory, rate limiting, the rebrand from dev codename to customer-facing name, and where I think this goes next, including the MCP-shaped future I keep circling.
Next up, Part 5: Production Realities. Caching, performance, memory, rate limiting, shipping, and what comes after.