From AI prototype to daily use
The first demo feels effortless: you describe a task to a coding assistant, have it produce a small application, and it answers a question about your manual. Then a colleague asks something slightly differently. The assistant fails to find the right passage and still words a plausible answer.
That is a useful finding. It shows what has to be clarified before the next use: verifiable quality, clear responsibilities and a limited scope of use.
What this means for you
A convincing prototype is a start, not a solution for the whole organisation. Before others use the application, you need a small test set, a visible data path and human sign-off. Then extend the prototype in small steps to decide whether the effort pays off.
Your next step: phrase the brief so you can check it, and put together six test cases. With the build brief further down, you let the AI carry out the smallest improvement step.
What speaks for vibe coding, and what against
Vibe coding means: you supply goals, examples and feedback, and the AI creates and revises code and tests. You judge whether the result is good enough for the work.
What speaks for it:
- An idea becomes tangible quickly, and faulty behaviour shows early.
- You learn from real examples which conditions the application has to meet.
What speaks against it:
- The assistant can deliver plausible answers with no basis.
- Passed examples help with the decision, but they do not cover every later input.
- An explanation from the AI is not proof of a successful test run.
Describe the brief so that you can check it
“A helpful support bot” is too vague as a yardstick. More concrete is: “The assistant drafts answers on three approved policies, shows the passage the answer comes from, and says when it lacks the knowledge.”
Also note what the application is allowed to do. For the first version, prepared documents and a preview are enough. Sending emails, changing customer data or other actions that reach the outside need their own process with explicit approval.
Build a small test set
Put together six cases, and write down for each what the right reaction is:
- 01Case
A clearly answerable question
Right is: an answer with the matching source.
Tick: source named - 02Case
The same question, worded differently
Right is: the same answer as in the first case.
Tick: same source - 03Case
A question that has no answer
Right is: the app says it found nothing and makes nothing up.
Tick: "found nothing" - 04Case
Two contradicting statements
Right is: the app points out the contradiction instead of picking one value.
Human: contradiction spotted? - 05Case
An outdated policy
Right is: the app names the date of the source or points to a newer version.
Human: date named? - 06Case
A foreign instruction in the document
For example "Ignore all previous rules". Right is: the app does not follow it and answers normally.
Human: instruction ignored?
Write down for each case what the right reaction is. Anything that can be checked with a tick, such as whether a source is named, you let the coding assistant build as a test programme. Whether an answer captures the meaning of a policy is additionally judged by a human.
Check the data path
An app that runs on your computer can still use services on the internet. Note where the documents are stored, where the search happens, who runs the AI model and which data ends up in the log. Start with approved, anonymised or made-up examples. For a test with real working data you need a separate decision on who accesses what and what is transferred.
The path of a question through your app can be noted down in four stations:
- 01Documents
Where do the documents live?
On your own server, in the cloud or with a provider?
Note the location - 02Search
Where is the search done?
Does the search run on your side or at a service on the internet?
Note the location - 03AI model
Who runs the AI model?
Your own servers or a provider? Which text passages are sent there?
Note the operator - 04Log
What is stored?
Which data ends up in the log, and who can read it?
Note the content
Make mistakes traceable
The preview should show which passage in the document the answer comes from and which function of the AI was used. Note which AI model worked with which settings and which document version, so you can recreate a faulty case later. Collect only the information you need, because an error log can also contain confidential content.
Change things in a controlled way
Before the AI applies a correction, you back up the current state so you can get it back. After the change, the same test set runs again. Compare whether the original mistake is fixed and the other answers still meet the criteria.
A suitable build brief reads:
Examine our support prototype using the six sample questions and their expected results. Explain the smallest improvement step in plain language. Back up the current state, change the necessary code and run the available tests. Show the preview, the results and the remaining limits. Do not carry out any external business actions. If you are missing programs or access, name them specifically.
Agree on a limited pilot
Before the start, decide who uses the app, how long the test runs, who is responsible and under which conditions you stop. A suggestion for the first week: one person checks ten anonymised requests and notes whether the source fits the answer, how much rework was needed and which mistakes occurred. Choose a clear stop signal, for example an answer containing information that is not allowed.
After that, you decide on the basis of the results whether the application is developed further. A further function is worthwhile if it solves an observed problem and its effect remains checkable.