How AI Budget Assistant uses AI: a function-calling agent, vision, voice, and data protection

In our product article we've already covered what AI Budget Assistant can do: photograph a receipt, dictate an expense by voice, ask the chat for budgeting advice — all without a single spreadsheet. That was the user's-eye view. Now let's come at it from the implementation side and see exactly how artificial intelligence is used here — and, just as importantly, what keeps it from doing anything silly.

Personal finances and AI are, above all, a question of trust. A generic chatbot on this topic easily says too much: it will confidently name a figure that doesn't exist, «recall» a transaction that never happened, or advise on something that has nothing to do with your budget. We built the system so the model doesn't work in a vacuum but on top of your real data, and so there is enough discipline around it: the AI key is kept on the server (not in the app), your data is fed to the model in isolation from the instructions, and wherever the model isn't needed, ordinary rules and mathematics do the work — faster, cheaper, and more predictable.

One idea runs through the whole text: AI here is a tool under control, not an oracle. Let's go in order: where it's used, how it's built, and where we deliberately did without it. Along the way — diagrams tidied up for a reader rather than an engineer.

Where the AI is

The app is a mobile client (built on Expo), while all the «smart» logic lives on the server. The client never calls OpenAI directly: it goes to our server, which then adds the key and the right context and calls the model. That way the key never leaves the server, and we can count quotas, sanitize data, and inject context all in one place.

flowchart TB
  subgraph Clients["How users connect"]
    APP["Mobile app<br/>(Expo)"]
    TG["Telegram bot"]
    WA["WhatsApp bot"]
  end
  subgraph Server["Server (NestJS) — the key lives here"]
    PROXY["AI proxy<br/>adds key + context"]
    RULES["Logic without AI<br/>anomalies · inflation index"]
  end
  OPENAI["OpenAI API"]
  subgraph Data["Storage"]
    PG["PostgreSQL"]
    RED["Redis · cache and limits"]
  end
  APP --> PROXY
  TG --> PROXY
  WA --> PROXY
  PROXY --> OPENAI
  PROXY --> PG
  PROXY --> RED
  RULES --> PG

Note two things. First, there's a single provider — OpenAI; no «zoo» of models is needed here. Second, right next to the AI on the server lives logic without AI — anomaly detection and the personal inflation index are computed by ordinary code. We'll come back to them separately.

One key, three model «speeds»

Unlike systems where you bring your own key, here the key belongs to the service. The user chooses not a provider but a model «speed» — in Settings, with a single switch that applies to all text and «vision» features at once.

flowchart LR
  REQ["AI request<br/>chat · receipt · suggestion"] --> Q{"Enough AI quota?"}
  Q -->|no| STOP["Refuse +<br/>suggest a plan"]
  Q -->|yes| MUL["Deduct quota<br/>× plan multiplier"]
  MUL --> RES["Model by «speed»<br/>user's choice"]
  RES --> F["fast<br/>gpt-4o-mini · ×0.75"]
  RES --> B["balanced default<br/>gpt-4o · ×1.0"]
  RES --> Qy["quality<br/>gpt-4.1 · ×1.5"]
  F --> CALL["Call OpenAI<br/>key on the server"]
  B --> CALL
  Qy --> CALL
  CALL --> ANS["Answer to the user"]

There are three speeds: fast (the gpt-4o-mini model, cheaper), balanced (gpt-4o, the default), and quality (gpt-4.1, pricier and more detailed). Each has its own cost multiplier that affects how much AI quota a single request spends: on the Free plan that's 5 AI requests per month, and a «fast» request costs 0.75 of a unit while a «quality» one costs 1.5. So the user controls the «cheaper / smarter» balance themselves, and costs stay in check. Voice recognition stands apart: it always goes through the fixed whisper-1 model and is not part of the «speed» choice.

The chat assistant as a function-calling agent

The chat isn't just «talking to a bot». Under the hood it's an agent with 11 functions (create an expense, add income, and so on): the model itself decides whether to answer with text or propose a concrete action, and vetted server code then carries it out.

flowchart TB
  MSG["User message"] --> MODEL["Model with 11 functions<br/>create expense, income…"]
  MODEL --> D{"What did the model return?"}
  D -->|plain answer| ANS["Text reply<br/>advice, analysis"]
  D -->|function call| ACT["Proposed action<br/>draft"]
  ACT --> ROLE{"User role?"}
  ROLE -->|viewer| BLOCK["Write blocked"]
  ROLE -->|owner / editor| CONF{"Confirm?"}
  CONF -->|yes| EXEC["Execute and save"]
  CONF -->|no| CANCEL["Cancel"]

An important detail — any change to your data goes through a confirmation. The model doesn't create an expense silently: it proposes an action, and the user confirms it (or cancels). And if a member of the account has the «viewer» role, the write is blocked before the action is even queued — both in the app and in the bots. In other words, the AI here helps you fill things in, but doesn't dispose of your data behind your back.

What the AI knows about you — and injection defence

To make its advice relevant, the assistant receives a slice of your finances: how much you've spent this month, your category limits, your top categories, and recent expenses. But this is exactly where a subtle risk hides — an attack through your own data (prompt injection): if a command for the model is hidden in an expense description, it could «hijack» the instructions. We close this off on two levels.

flowchart TB
  subgraph Ctx["User context"]
    C1["Monthly spending"]
    C2["Category limits"]
    C3["Top categories"]
    C4["Recent expenses"]
  end
  Ctx --> SAN["Sanitize text fields"]
  SAN --> BLOCK["Isolated data block<br/>--- USER DATA ---"]
  INSTR["Assistant instructions<br/>separate"] --> MODEL["Model"]
  BLOCK --> MODEL
  MODEL --> NOTE["Data is data,<br/>not commands"]

First, all user strings (expense descriptions, project, tag, category, and goal names) are sanitized before being placed into the prompt. Second, the context is passed to the model as an isolated data block, wrapped in service markers (something like «--- USER DATA ---» … «--- END OF DATA ---»), so the model treats it as data, not as instructions. Even a free-form caption on a scanned receipt is length-limited and passed as a passive note rather than a command.

A receipt from a photo: vision

The fastest way to log an expense is to photograph the receipt. Recognition is handled by a «vision» model (the same one you picked by «speed»): it pulls the line items, amounts, date, and even the store address out of the photo.

flowchart LR
  PHOTO["Receipt photo"] --> OCR["Vision recognition<br/>selected model"]
  OCR --> ITEMS["Line items: product · price ·<br/>date · store · address"]
  ITEMS --> EXP["Expense draft<br/>please confirm"]
  ITEMS --> CANON["Canonical product name<br/>for the inflation index"]
  ITEMS --> ADDR["Store address"]
  ADDR --> GEO["Geocoding: OpenStreetMap<br/>no AI, cached"]
  GEO --> MAP["Point on the map"]

Then comes the careful work with details. Each line item is given a canonical product name — needed so that prices can later be compared honestly over time (more on that below). And the store address becomes a point on the map via an open geocoding service (OpenStreetMap/Nominatim) — with no AI at all, cached and «quiet» on failure. A good example that not every task should be handed to a model.

Voice: dictate an expense

You don't have to type an expense — you can say it out loud. A voice message is transcribed by the whisper-1 model, after which the text is handled like an ordinary phrase — by the same expense parsing as keyboard input. This works both in the app and in the messengers.

AI in Telegram and WhatsApp

AI Budget Assistant doesn't live only in the app. It has full-featured bots in Telegram and WhatsApp, and — importantly for the architecture — they reuse the very same AI services: the same chat agent, the same voice transcription, the same receipt recognition. Not three different «brains», but one, reachable from different channels.

flowchart TB
  APP["App"] --> S
  TG["Telegram"] --> S
  WA["WhatsApp"] --> S
  subgraph S["Shared AI services on the server"]
    CHAT["Chat agent"]
    VOICE["Voice → text<br/>whisper-1"]
    OCR["Receipts — vision"]
  end
  S --> OPENAI["OpenAI"]

A nice touch: in a conversation the bot can recognise an account name in the text and answer specifically about it, without switching your main account. Just write something like «show expenses in Family» and the bot will understand which account you mean.

Smart suggestions: categories, tags, projects

A good helper doesn't make you fill everything in by hand. So when you enter an expense, the system suggests a category, tags, a matching project, and even a split of the amount across several categories. But here too the model isn't always involved.

flowchart LR
  E["New expense"] --> H{"Seen something<br/>similar before?"}
  H -->|yes| FAST["Instant suggestion<br/>no model"]
  H -->|no| AI["AI / embeddings"]
  AI --> S1["Category"]
  AI --> S2["Tags"]
  AI --> S3["Project by meaning"]
  AI --> S4["Split across categories"]

The logic is simple and thrifty: history first. If a similar expense has been seen before, the suggestion is served instantly, with no call to the model. And only for unfamiliar cases does the AI kick in, while for matching «by meaning» (for instance, which project an expense belongs to) embeddings — vector representations of text — are used. That way frequent operations spend neither AI quota nor time.

Insights, stories, and the portfolio

On top of your data, the AI can not only answer questions but also spot patterns on its own. It assembles insight cards (where money leaks away, what has changed) and even narrative «spending stories» — a coherent account instead of dry figures.

flowchart LR
  DATA["Your finances"] --> AI["The model you chose"]
  AI --> CARDS["Insight cards<br/>spending patterns"]
  AI --> STORY["Spending story<br/>narrative summary"]
  PORT["Investment portfolio"] --> AIP["Portfolio insights<br/>Pro+ tier, 24h cache"]
  AIP --> RISK["Concentration risk,<br/>underperformance, fees…"]

The investment portfolio is a story of its own. Here the AI analyses the composition of your holdings and produces portfolio insights: concentration risk in a single asset, a skew by type, lagging behind a benchmark, the impact of fees, and other signals — with clear «warning / critical» thresholds. This capability belongs to the paid tier (Pro+), costs a bit more than an ordinary request, and is cached for a day so the same thing isn't recomputed.

Where the AI is deliberately NOT used

This is perhaps the most underrated section. Not everything that looks «smart» should go to a model — sometimes ordinary code is more honest, faster, and cheaper. We've done two notable things this way.

flowchart TB
  EXP["Expense saved"] --> RULES["4 rules — no AI"]
  RULES --> R1["Category spike<br/>+30% vs average"]
  RULES --> R2["Subscription price up<br/>over 10%"]
  RULES --> R3["Duplicate charge<br/>±1 day"]
  RULES --> R4["Looks like a recurring payment"]
  R1 --> FEED["Alert feed<br/>dedup protection"]
  R2 --> FEED
  R3 --> FEED
  R4 --> FEED
  FEED --> PUSH["Push — max 3 per day"]

Anomaly detection runs on four simple rules that fire right when an expense is saved: a category spike (a rise of 30%+ over the average), a subscription that got pricier (a price rise of more than 10%), a duplicate charge (the same payee and amount within a day), and «looks like a recurring payment». Each signal lands in a deduplicated feed, and to avoid nagging, no more than three pushes are sent per day. No model — only deterministic rules that always fire the same way.

The personal inflation index is more honest still. It shows how the cost of your very own «shopping basket» changes, and is computed with ordinary mathematics (a Laspeyres index) over the line items from your receipts — without a single call to AI. The model plays only an indirect part here: it once recognised a receipt, and arithmetic does the rest. Cheap, transparent, and reproducible.

Conclusion

Put it all together and the same simple thought emerges as in our other products: value comes not from «a bigger model» but from careful engineering around it. AI that works on top of your real data rather than inventing it. A key that stays on the server rather than leaking into the app. Data isolated from instructions. A confirmation before any write. And the sober decision not to summon a model where rules and mathematics will do.

It's exactly this discipline that turns a talkative model into an assistant you can trust with your budget — without fear that it will paint in an extra expense for you. You can try AI Budget Assistant at ai-budget.pl, on Google Play, or build it yourself from the sources on GitHub. And if you need a similar AI helper embedded into your own product and data, write to us: [email protected].

Which model does what

Finally — a summary of which model handles which task. One thing to underline separately: anomaly detection and the personal inflation index work without a model — on rules and mathematics, so they're not in the table.