Quality and tests

We compare how accurate the AI is against a simple rule-based approach, check that no numbers are invented and test the data itself. The tests are open and can be rerun. Last updated: 12 Oct 2026, 16:20 (Baku time).

Understanding intent (location)

100%

rule: 85%

Legal risk: recall

100%

rule: 58%

Number grounding

99.8%

624 numbers checked

Results meeting constraints

100%

10/10 queries

Sample sizes are small (10–30 cases); results are indicative only.

1. Intent: AI vs keyword rule

27 queries in Azerbaijani (3 ambiguous, 2 misspelt or colloquial). Accuracy by field.

Goal

AI (Gemini)

100%

Keyword rule

96%

Property type

AI (Gemini)

96%

Keyword rule

100%

Location

AI (Gemini)

100%

Keyword rule

85%

Constraints

AI (Gemini)

89%

Keyword rule

100%

Budget

AI (Gemini)

100%

Keyword rule

93%

English queries

11 queries in English (same schema, same checks).

Goal

AI (Gemini)

100%

Keyword rule

100%

Property type

AI (Gemini)

100%

Keyword rule

100%

Location

AI (Gemini)

100%

Keyword rule

91%

Constraints

AI (Gemini)

91%

Keyword rule

100%

Budget

AI (Gemini)

100%

Keyword rule

82%

2. Legal check: risk in listings

18 listing texts: 6 clean, 12 risky (including a hidden risk: “sənədlər sonra düzələcək”, documents to be sorted later).

Recall (how many risky listings were found)

AI (Gemini)

100%

Keyword rule

58%

Precision (how many warnings were correct)

AI (Gemini)

86%

Keyword rule

100%

On the site the two work together: any phrase the rule finds is always added to the AI's list, so known phrases are never missed.

3. Are numbers invented?

Every number in 29 AI texts was extracted and looked up (±1%) in the JSON given to the model.

Found in JSON

99.8%

target

100%

Not found: 1. Examples: az-xankendi → 355

English profiles: 29 texts, 747 numbers, 98.8% found in the JSON.

4. Meeting constraints (retrieval)

Query (hard constraints)	Matching / returned	Result
Nərimanov + metro ≤800 m	5 / 5	✓ passed
Yasamal + school ≤500 m	5 / 5	✓ passed
Bakı + metro ≤500 m + kindergarten ≤600 m	5 / 5	✓ passed
Binəqədi + park ≤400 m	5 / 5	✓ passed
Bakı + ≤2000 AZN/m²	5 / 5	✓ passed
Outside Baku + mountainous	5 / 5	✓ passed
Coastal district	5 / 5	✓ passed
Xətai + metro ≤1000 m + school ≤700 m	5 / 5	✓ passed
Gəncə + school ≤800 m	5 / 5	✓ passed
Nəsimi + metro ≤600 m + park ≤500 m	4 / 4	✓ passed

5. Data checks

Result	Check	Detail
✓	75 districts/cities present	75
✓	All 12 Baku districts present	Nizami, Sabunçu, Xətai, Pirallahı, Suraxanı, Xəzər, Qaradağ, Yasamal, Səbail, Nərimanov, Nəsimi, Binəqədi
✓	Every area has a population (SSC)	all have one
✓	Sum of Baku district populations = Baku (±2%)	2356200 vs 2356200
✓	No NaN or negative values	none
✓	No district holds more than 25% of the country's POIs (except Baku)	max: 6.0%
✓	Per-10,000-resident figures are plausible (≤10× median)	all within range
✓	Areas with a population–OSM mismatch are flagged	Xankəndi, Ağdərə, Xocalı
✓	No area scores 99 on 4+ indices	none
✓	PM2.5, temperature, density and paved % within plausible limits	all
✓	Prices within plausible limits (200–20,000 AZN/m²)	101 prices checked
✓	Metro: 27 operating stations (from official exits)	Bakı 27, opendata 27 stations / 98 exits
✓	opendata.az: every dataset has records	settlements 4589, gazetteer 4589, rail_regions 87, roads 60, poi_areas 14, metro_pax 27
✓	Every settlement (PARENT_ID) maps to a district	4589 settlements
✓	No NaN in the new official metrics	none
✓	Gazetteer: lowercase village names resolve; common words are not treated as villages	5 checks
✓	H3 compute grid: every cell lies inside its own district	3700 cells checked
✓	H3 compute grid: sum of cells ≈ district area (±3%)	max difference 1.60%
✓	Every neighbourhood lies inside its own district	1125 neighbourhoods checked
✓	Neighbourhoods do not overlap (simplification allowance ≤0.5%)	14 districts
✓	Total neighbourhood area ≈ district area (±3%)	max difference 0.11%
✓	The place in each neighbourhood name lies inside its own district	1125 names checked
✓	Every neighbourhood has a name	1125 names
✓	Share of areas with low OSM coverage	17/87 (20%)
✓	Share of real (published) prices	61/101

6. Our mistakes and how we fixed them

Xankəndi came first for every query

Input:: Any persona at country level (e.g. 'family with children')
Wrong result:: Xankəndi scored 99 on every per-capita index and was ranked first; the AI wrote every answer about Xankəndi.
Cause:: A time mismatch in the data: OSM holds 409 features from the pre-2023 city of ~55,000 residents, while DSK gives a 2026 population of 4,400, i.e. 929 features per 10,000 residents (median 19.5, 47.7×).
Fix:: General rule: if the feature/resident ratio exceeds 10× the median, per-capita indicators become null, the panel shows a warning and an unknown index counts as 30 in the fit score. The data-sanity eval checks this on every build (Xankəndi, Xocalı and Ağdərə are flagged).

A metro station that does not exist was recommended

Input:: “Nərimanovda metroya yaxın ev lazımdır” (I need a home near the metro in Nərimanov)
Wrong result:: The third recommendation was “Nərimanov · 20 Sentyabr metro area”, a station that is not open yet.
Cause:: In OSM the station=subway tag is also used for a depot (Dərnəgül depot), an exit (28 May exit 2) and planned stations (Y14–Y16, 20 Sentyabr, Məhəmməd Hadi): OSM showed 33 'stations', the real number is 27.
Fix:: OSM stations are checked against the list of 27 operating stations, name variants are normalised (“Metro elmlər” → “Elmlər Akademiyası”) and the rest are dropped. data-sanity: OSM 26 / official 27 (Memar Əcəmi-2 is at the same location).

Numbers in the AI profiles were out of date

Input:: Grounding eval: 29 AI profiles, 402 numbers (first measurement)
Wrong result:: 25 numbers (6.2%) were not found in the input JSON, e.g. 'index 57' for Füzuli.
Cause:: The profiles were written before the Xankəndi fix; the fix changed every district's percentiles but the text stayed the same.
Fix:: Each profile stores a hash of its input data; when the data changes the profile is marked stale and rewritten (--refresh-stale). Result: grounding 93.8% → 99.8% (1 of 639 numbers not found).

The confidence score came out as NaN

Input:: “Bakıdan kənarda dağ havası, həyət evi” (mountain air outside Baku, a house with a yard) — districts without prices
Wrong result:: Recommendation confidence: NaN
Cause:: Operator precedence: s + X ?? 0 was evaluated as (s + X) ?? 0; with no price, undefined + number = NaN.
Fix:: Parentheses added; the retrieval eval checks that the score is between 0 and 100 on 10 queries.

Azerbaijani letters were lost when falling back to a weaker model

Input:: Lerik profile (switch to the flash-lite model when the free limit ran out)
Wrong result:: “zengin tebieti ve mehdud infrastrukturu olan dagliq erazi”: the letters “ə”, “ğ” and “ı” are missing.
Cause:: On a 429 limit, the fallback model chain reached the weaker flash-lite model.
Fix:: A profile quality check (length, ≥6 numbers, banned phrases, Azerbaijani letters); failing profiles are rewritten.

The model invented place IDs that do not exist

Input:: “nerimanovda kiraye ev axtariram” (looking for a rental in Nərimanov), “Səbaildə investisiya üçün mənzil” (a flat in Səbail as an investment)
Wrong result:: The AI returned non-existent IDs such as “az-nerimanov” and “az-sebail”, sometimes adding “az-baki” next to the district, so the search widened to all of Baku.
Cause:: The model wrote the district in the country-level ID format and also added the parent city.
Fix:: Deterministic normalisation: IDs not in the catalogue are dropped, az-X → bk-X (only if a Baku district exists) and “az-baki” is removed when a district is named. Location accuracy 75% → 95%.

7. Comparison with the current approach

The same task (“pick 3 areas in Nərimanov near the metro with a school and find their prices”): by hand with bina.az + Google Maps vs Məskən.

Participant	By hand, min	Məskən, min
Participant 1	being measured	being measured
Participant 2	being measured	being measured
Participant 3	being measured	being measured

User testing is under way; results and quotes will be added here. We make no claims until we have the numbers.

8. Cost

Model (src/lib/ai.ts): gemini-3.7-flash; fallback on rate limit or overload: gemini-3.8-flash, gemini-3.6-flash, gemini-3.5-flash, gemini-flash-latest, gemini-3.5-flash-lite.
Measured average: 21,032 input + 425 output tokens per call (141 calls)
Session (~4 AI calls): $0.03 (gemini-3.7-flash price) … $0.14 (gemini-3.5-flash price, cautious estimate)
1,000 users: $34.74 … $141.49
Price sources: gemini-3.7-flash (primary model): $0.375/$1.88 per 1M (pricepertoken.com (no Google confirmation found)); gemini-3.5-flash (fallback model): $1.5/$9 per 1M (benchlm.ai / morphllm.com). The free Gemini tier is enough for the demo; the cache makes repeat queries free. Hosting: Vercel Hobby (free), the data is static.

9. Services: OSM vs the official list

OSM counts for Baku's districts, Sumqayıt and Abşeron (14 areas) are compared with the opendata.az list (Ministry of Ecology and Natural Resources). The sources are not added together. 1,593 places have the status “Checking”; 924 CUSTOMIZABLE_POINT entries are ignored.

Category	OSM	opendata.az
School	1,099	655
Kindergarten	682	331
Hospital	690	527
Pharmacy	645	482
Grocery shop	4,358	1,819
Bank / ATM	1,004	644
Park	3,338	79

Biggest gaps

Area	Category	OSM / opendata.az
Abşeron	Grocery shop	754 / 50
Xəzər	Grocery shop	588 / 11
Sabunçu	Grocery shop	581 / 83
Suraxanı	Grocery shop	431 / 48
Xəzər	Park	375 / 1
Binəqədi	Grocery shop	471 / 99
Sumqayıt	Park	377 / 21
Sabunçu	Park	331 / 1

OSM categories are broader (e.g. grocery kiosks, small green spaces) and the official list may be incomplete. A gap does not show which source is right; it shows where checking is needed.