How It Works — Guess Who

What Is This?

A real-time multiplayer Guess Who game with a twist: a hybrid verification engine that combines instant database lookups for factual traits with a local Vision-Language Model (VLM) for subjective "vibe check" questions like "Does this person look like a barista?"

All AI inference runs locally — no external API calls, no cloud dependencies.

How a Game Works

1Enter your name and get matched with another player via FIFO matchmaking.

2Both players see 24 randomly selected characters and secretly pick one as their identity.

3Take turns asking yes/no questions or guessing the opponent's character.

4Questions are evaluated against the opponent's secret character — factual traits resolve instantly, subjective ones go through the AI model.

5Eliminate characters from your board based on answers. Your opponent sees your eliminations in real time.

6Correctly guess the opponent's character to win!

The Hybrid Verification Engine

When you ask a question, the system decides how to answer it:

Question: "Do they have brown hair?" │ ▼ ┌─────────────────────────┐ │ Check character attributes │ │ (DB key/value lookup) │ └────────────┬────────────┘ │ Match found? ──YES──▶ Instant verdict (Yes/No) │ NO │ ▼ ┌─────────────────────────┐ │ Send to VLM (Ollama) │ │ image + attributes + question │ └────────────┬────────────┘ │ ▼ Yes / No / Inconclusive

Factual questions (hair color, glasses, hat) resolve instantly from the database.

Subjective questions ("looks friendly", "seems athletic") get evaluated by the AI model looking at the character's image and metadata.

If the model can't decide, you get Inconclusive — the round still completes, you just don't get useful info from that question.

Tech Stack

BackendScala 3 + ZIO

HTTP/WebSocketZIO-HTTP + Tapir

FrontendSvelteKit + TypeScript

DatabasePostgreSQL + Skunk

AI ModelOllama (qwen2.5vl:3b)

Reverse ProxyCaddy (auto TLS)

Architecture

┌──────────┐ WebSocket ┌──────────────┐ │ Browser │◀───────────▶│ ZIO-HTTP │ └──────────┘ │ (Game Loop) │ └──────┬───────┘ │ ┌────────────┼────────────┐ ▼ ▼ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ PostgreSQL│ │ Ollama │ │ Caddy │ │ (chars, │ │ (VLM) │ │ (TLS + │ │ history) │ │ │ │ static) │ └──────────┘ └──────────┘ └──────────┘

WebSocket — persistent connection for the entire game session; real-time turn updates
PostgreSQL — stores characters with flexible JSONB attributes, plus game history
Ollama — runs locally for subjective question evaluation; CPU inference with quantized models
Caddy — automatic HTTPS via Let's Encrypt, reverse proxy to the app

WebSocket Protocol

The game uses a tagged-union message protocol over WebSocket:

Client → Server

SelectCharacter — pick your secret character
SubmitQuestion — ask a yes/no question
GuessCharacter — guess the opponent's character
EliminateCharacter — flip a card on your board

Server → Client

GameStart — match found, here are your 24 characters
TurnStart — whose turn it is
RoundResult — Yes / No / Inconclusive
GuessResult — correct or wrong guess
GameOver — winner announced

Play It

The game is live at mosely.xyz. Open two tabs, enter different names, and you'll be matched together.