# Guide Detection / Generation — Audit

**Date:** 2026-06-13
**Auditor:** overnight session (Opus orchestrator)
**Data source:** live `tenant_internal` schema via app context (read-only). Collector: `tmp/audit_collect.py`, sweep: `tmp/sweep.py`.

## 1. Method

- Loaded the production `EmbeddingIndex` exactly as startup does (`load_from_db` + `load_doc_pages_from_db`, schema `tenant_internal`).
- Index size: see `tmp/audit-data.json` → `stats.index_count`.
- Pulled `count_guide_candidates_batch` (the /products badge source) and `get_guide_candidates` (the per-product API source) for the 6 products with the most strong-quality threads.
- Read the **one** existing `GeneratedGuide` and the **full message bodies of 14 candidate threads** (top clusters across products) to judge each on documentation merit.

## 2. Headline finding — candidacy is massively over-counted

| Product | strong threads | /products badge (batch) | get_guide_candidates |
|---|---:|---:|---:|
| Estimated Delivery Date V3 | 1457 | **148** | 575 |
| Pixel Plus for Facebook | 1408 | **172** | 604 |
| Product Videos | 1324 | **151** | 609 |
| Products Feed (Facebook) | 1115 | **141** | 547 |
| Conversion Pixel Tracking | 325 | **38** | 106 |
| WhatsApp Contact | 143 | **22** | 85 |

Two distinct bugs:

1. **`get_guide_candidates` has no `min_cluster_size` filter** — it returns *every* cluster seed including singletons, so a product shows 500–600 "candidates" (one per near-unique thread). It also uses threshold `0.82`.
2. **`count_guide_candidates_batch` uses threshold `0.78` + `min_cluster_size=2`** — a recurring "topic" only needs to happen twice, and 0.78 on short generic `Problem: X. Solution: Y.` summaries pairs up many *unrelated* threads. Result: 140–172 "possible guides" per product, almost all noise.

The summaries embed well enough to surface the **big** recurring topics, but the long tail of size-2/size-3 clusters is junk.

## 3. Quality of the candidates themselves

Even the **largest** clusters (the best-ranked candidates) are mostly **support debugging sessions**, not documentation material. Recurring anti-patterns that make a thread a *bad* guide source:

- resolution was "send me BO/FTP access" then fixed privately (no reusable steps in the thread);
- resolution was "update to the latest version" (version-specific bug, not documentation);
- thread mixes several unrelated problems;
- contains live customer credentials / personal data in the body.

Only threads where the **answer is a self-contained, reusable instruction** make sensible guides.

### Per-candidate verdicts (14 threads read in full)

| Thread | Product | Subject | Verdict | Reason |
|---|---|---|---|---|
| t3575 | Est. Delivery | Bug affichage (carrier info not shown) | ✅ GOOD | Clean reusable fix: enable advanced options → "force search country". Recurs (see t464). |
| t464 | Est. Delivery | Only works when logged in | ✅ GOOD | Same "force country search" fix — genuinely recurring, documentable. |
| t5704 | Est. Delivery | Language / multilanguage locales | ✅ GOOD | Reusable how-to on server locales + translation. |
| t1328 | Est. Delivery | Show delivery on home/category pages | ✅ GOOD | Reusable template-hook placement guide. |
| t5310 | WhatsApp | Clickable WhatsApp icon placement | 🟡 OK | Config how-to, usable but thread is thin. |
| t385 | Est. Delivery | Out-of-stock delivery days not showing | 🟡 OK | Real topic ("configure delivery days for out-of-stock") but thread is a BO debugging session. |
| t1142 | Est. Delivery | "No configuro bien" | 🟡 OK | Common config confusion; thread messy, needs heavy rewriting. |
| t604 | Est. Delivery | Feature request + save bug | ❌ POOR | Mixed feature request + version bug; not a how-to. |
| t5108 | Est. Delivery | Vacation days bug | ❌ POOR | Resolved by "I updated the module" — version bug. |
| t357 | Est. Delivery | Delay update + update impossible | ❌ POOR | Debugging session, BO access, no reusable answer. |
| t333 | Est. Delivery | Backend display bug | ❌ POOR | Resolved by sending a patched zip — version bug. |
| t544 | Est. Delivery | "problems when using the plugin" | ❌ POOR | BO/FTP debugging, **contains live credentials**. |
| t1479 | Est. Delivery | Compatibility SQL error (1.7.5.2) | ❌ POOR | BO/FTP debugging, **contains live credentials**. |
| t173 | Pixel Plus | Meta says invalid parameters | ❌ POOR | "Update the module" + private session; no reusable steps. |
| t5325 | Pixel Plus | "Problema con el modulo" | ❌ POOR | Multiple unrelated problems, BO debugging. |

**Precision of current top-of-list candidates ≈ 4 good / 14 ≈ ~30%**, and that is the *best* slice; the long tail (size-2 clusters) is far worse.

### The one existing GeneratedGuide

- **"Troubleshooting the Products Alert Module Visibility"** (ticket 4752, 3419 chars). Well-formed Docusaurus MDX (frontmatter, intro, steps, conclusion). This is the *target quality*. Note it was generated for a product not in the top-6 list, and reads well — so the generator works when fed a clean case; the problem is candidate *selection* and *grounding*.

## 4. Clustering parameter sweep (counts per product)

`tmp/sweep.py` output, candidate count = clusters with size ≥ min:

| Product | strong | 0.78/m2 (current) | 0.85/m4 | 0.86/m5 | 0.88/m5 | **0.88/m6** | 0.90/m6 |
|---|---:|---:|---:|---:|---:|---:|---:|
| Est. Delivery | 1457 | 149 | 56 | 27 | 23 | **19** | 10 |
| Pixel Plus | 1408 | 172 | 59 | 33 | 18 | **12** | 5 |
| Product Videos | 1324 | 151 | 52 | 27 | 22 | **15** | 7 |
| Products Feed | 1115 | 142 | 37 | 26 | 11 | **6** | 1 |
| Conversion Pixel | 325 | 38 | 14 | 10 | 12 | **12** | 6 |
| WhatsApp | 143 | 22 | 4 | 3 | 0 | **0** | 0 |

**Chosen target: `cluster_threshold = 0.88`, `min_cluster_size = 6`.**
- Every product lands **≤ 19** candidates (success criterion "<~20 per product" met).
- A topic must recur **6+ times** at high (0.88) similarity to qualify → only genuinely repeated issues survive, which is exactly the precision we want given §3.
- WhatsApp Contact → 0 is acceptable: with only 143 threads it has no topic recurring 6× at 0.88; better no badge than a noisy one. (Recall is intentionally traded for precision.)

Re-embedding on fuller thread text (instead of the 1–2 sentence summary) would further sharpen clusters, but requires re-embedding ~6k threads (OpenAI cost + a migration) and is out of scope for this pass. Threshold+min-size tuning already achieves the <20 target. Recorded as a future improvement.

## 5. Rework decisions (implemented this session)

- **(a) Decouple candidacy from real-time classification.** Remove `could_be_guide`/`guide_title` from the classify prompt and the `if not could_be_guide: return 400` gate on manual generation. The per-ticket "Guide" button now keys off a server-computed `is_guide_candidate = (ticket.status == 'closed' AND thread embedding_quality == 'strong')`.
- **(b) Tune clustering.** `count_guide_candidates_batch` and `get_guide_candidates` both move to threshold `0.88` / `min_cluster_size 6`. `get_guide_candidates` gains the missing min-size filter.
- **(c) Improve generation.** `generate_guide_from_summary` max_tokens `800 → 1500` and grounds the article in the **full thread (last N messages)**, not just the 1-sentence summary.
- **(d) Verify.** Regenerate candidate lists for 2–3 products and confirm counts drop into the plausible range and the surviving candidates align with the GOOD/OK verdicts above.

## 6. Post-rework verification

Run via `tmp/verify.py` (fresh process = new code; in-process Flask test client, systemd server untouched).

### Candidate counts: before → after

| Product | strong | badge before | badge after | detail before | detail after |
|---|---:|---:|---:|---:|---:|
| Estimated Delivery V3 | 1457 | 148 | **19** | 575 | **19** |
| Pixel Plus | 1408 | 172 | **9** | 604 | **16** |
| Product Videos | 1324 | 151 | **15** | 609 | **14** |
| Products Feed | 1115 | 141 | **6** | 547 | **5** |
| Conversion Pixel | 325 | 38 | **11** | 106 | **5** |
| WhatsApp Contact | 143 | 22 | **0** | 85 | **1** |

Every product now ≤ 19 (criterion "<~20" met). The batch badge and per-product detail list are now in the same ballpark (they differ slightly because the batch path additionally dedups against existing DocPages; the detail path only dedups against existing guides).

### Precision check (new top candidates vs §3 verdicts)

The surviving candidates are coherent, high-volume recurring topics — exactly the documentable categories:
- **Estimated Delivery:** "module not displaying delivery time / not displaying on product page" (clusters of 56, 17, 13, 12…) — the GOOD "display/force-country" family from §3.
- **Pixel Plus:** "pixel triggering incorrectly / CAPI not working / pixel code not loading" (18, 14, 13…) — coherent tracking-setup topics.
- **Product Videos:** "videos not displaying / can't add video" (25, 12, 10…) — coherent display topics.

The individually-messy threads (t173, t5325) now appear only as members of larger coherent clusters, representing a real recurring topic rather than standalone noise. Precision of the surfaced list is visibly higher than the 30% measured in §3.

### Endpoint + compile checks

```
py_compile main.py / ai_service.py / embedding_service.py  -> OK
POST /login   -> 302 (auth ok)
GET  /products -> 200
GET  /guides   -> 200
GET  /ticket/101 -> 200
```

### Manual generation gate

The `if not could_be_guide: return 400` gate on `/api/suggestion/<id>/generate-guide` is removed; `could_be_guide`/`guide_title` are gone from the classify prompt. The per-ticket "Guide" button now keys off server-computed `window._guideCandidate` (= ticket closed + embedding_quality strong) AND the presence of an AI suggestion. Generation grounds the article in the last 12 thread messages with max_tokens 1500.