Add a table-driven test covering the markdown link destination check
before replacing the govalidator dependency with stdlib logic.
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
The search term was formatted into LOWER(%s) and passed as the *value* of the
LIKE, so the function name ended up inside the pattern:
slug_name LIKE '%LOWER(coco)%'
That can never match. Only the display_name clause did any work, and LIKE is
case-sensitive on Postgres, so searching a tag by the name it is written in
returns nothing:
slug_name=Coco -> matches
slug_name=coco -> no match
Tags are lower case by convention, so lower case is what users type, and the
filter appears to report that no such tag exists.
Lower both sides instead. The term normalisation is extracted so it can be
covered by a test without a database.
Replace OptionalBool with an explicit require_email_verification value for the login settings save request while keeping legacy read defaults intact.
Use positive RequireEmailVerification naming through the registration flow and inline the site setting mapping.
Add validation, save-path, and registration coverage for the explicit email verification setting.
The ownership check added to AcceptAnswer compared the answer's QuestionID
against the request's QuestionID directly. When short links are enabled,
answerRepo.GetByID re-encodes QuestionID to its short form while the
controller de-shorts req.QuestionID to its long form, so the two encodings
of the same question never matched and every accept returned "Answer do not
found". Normalize both ids via uid.DeShortID before comparing, preserving the
privilege-escalation guard for answers that truly belong to another question.
# fix: avoid `topic` fallback for non-Latin titles via pragmatic ASCII
transliteration
> **Scope update (in response to review):** this PR is intentionally
broader than its original "Arabic-only" framing. The implementation
changes URL slug generation for **every non-Latin, non-CJK script** that
`slugify` previously stripped — see *Scope* below for the explicit list.
The goal is *not* linguistically correct romanization; it is "avoid
collapsing to `/topic` by producing a usable ASCII slug."
## What this PR is (and isn't)
**Goal:** when a question title contains characters outside Basic Latin
/ Latin Extended / CJK Han, generate a URL slug that is a deterministic
ASCII approximation instead of letting `slugify` strip everything and
falling back to the literal `"topic"`.
**Non-goal:** this is *not* a linguistically correct multi-language
romanizer. The output is a machine-acceptable ASCII slug, not what a
native speaker would choose. For example, `こんにちは` → `konnichiha` (not
the more natural `kon'nichiwa`), `ไทย` → `aithy` (not `thai`). Treat the
slug as an opaque, stable, indexable identifier — the
path-after-`/questions/<id>/` is for SEO and shareability, the canonical
reference is always the ID.
## The bug
Pure non-Latin titles previously got stripped by `slugify.Slugify`, hit
the empty-result fallback in `htmltext.UrlTitle`, and collapsed to the
literal slug `"topic"`. On a live multilingual site, every Arabic / Thai
/ Japanese-hiragana / Korean / Hebrew / Cyrillic question ended up at
`/questions/<id>/topic`.
## The fix
`UrlTitle()` gets a `convertNonLatin` pre-step that mirrors the existing
`convertChinese` pre-step pattern, using
`github.com/mozillazg/go-unidecode` (same author as `go-pinyin` already
in the repo, to minimise new-dep friction).
```
UrlTitle(title)
→ convertChinese(title) // pre-existing: Han-block → pinyin
→ convertNonLatin(title) // NEW: detect non-Latin letters → unidecode to ASCII
→ clearEmoji / slugify / url.QueryEscape / cutLongTitle (unchanged)
```
The non-Latin detector skips ASCII, Latin-1 Supplement, Latin
Extended-A/B, and CJK Han. Inputs that hit none of those non-Latin
letter categories short-circuit and return unchanged, so Latin-only and
Chinese-only inputs remain byte-identical (pinned by tests).
## Scope — what scripts are affected
This PR changes behavior for **any** title containing letters in scripts
that `slugify` doesn't handle. Confirmed by tests in
`pkg/htmltext/htmltext_test.go`:
| Script | Example title | Before | After |
| --- | --- | --- | --- |
| Arabic | `كيف حالك` | `topic` | `kyf-hlk` |
| Mixed Latin + Arabic | `مرحبا hello` | `hello` | `mrhb-hello` |
| Thai | `ไทย ไทย` | `topic` | `aithy-aithy` |
| Japanese hiragana | `こんにちは` | `topic` | `konnichiha` |
| Korean | `안녕하세요` | `topic` | `annyeonghaseyo` |
| Hebrew | `שלום עולם` | `topic` | `shlvm-vlm` |
| Cyrillic | `Привет мир` | `topic` | `privet-mir` |
**Unchanged:**
| Case | Behavior |
| --- | --- |
| Pure Latin (`hello world`) | unchanged → `hello-world` |
| Pure Chinese (`这是一个，标题，title`) | unchanged → `zhe-shi-yi-ge-biao-ti`
(pinyin path) |
| Japanese with Han-block kanji (`日本`) | unchanged → `ri-ben` (caught by
pre-existing pinyin path; treated as Chinese reading, not Japanese — a
pre-existing limitation, **not** introduced by this PR) |
| Emoji only (`😂😂😂`) | unchanged → `topic` |
| Empty / whitespace | unchanged → `topic` |
## Transliteration quality — explicit acknowledgement
`go-unidecode` is a generic Unicode → ASCII approximation. It is **not**
a per-language romanization library. Specifically:
- It will pick *one* approximation per codepoint regardless of language
context. `ใ` → `ai` (Thai romanization is `i` or `ai` depending on
standard), `한` → `han`, `語` → `Yu` (Chinese pinyin reading even when
used in Japanese), etc.
- The result is *good enough* to be a stable, URL-safe,
human-recognizable handle, but speakers of the source language will not
consider it "correct."
- It is deterministic, so the same title always produces the same slug —
important since `url_title` is recomputed on every request.
If maintainers prefer to scope this PR more narrowly (e.g. Arabic only,
and reject Thai/Hebrew/Cyrillic/etc.), the detector in
`containsNonLatin` can be tightened to specific Unicode blocks — but
that means the other scripts continue to collapse to `topic`, which is
the bug we're trying to fix. I'd argue the broader fix is preferable to
a piecemeal one, but happy to narrow if you want.
## Live deployment / real-world verification
This patch has been running in production on
**[ask.namasoft.com](https://ask.namasoft.com)** (an Apache Answer
instance we operate) since deployment, built directly from this branch
via `docker compose build`. The site hosts Arabic-language questions, so
the fix exercises the affected code path on every page load.
Sample question URL on the deployed instance:
> `https://ask.namasoft.com/questions/10010000000000115`
The slug in the URL is the transliterated Arabic title rather than
`topic`. No data migration was needed since `url_title` is computed on
every request from `Title` and never persisted (see *Why this is safe to
ship* below).
## Admin-configurable
The transliteration is gated by a package-level `atomic.Bool` (default
**on**, since the current behavior is objectively broken for affected
users):
- `htmltext.SetTransliterateNonLatin(enabled bool)`
- `htmltext.IsTransliterateNonLatinEnabled() bool`
This is deliberately the minimum surface needed to satisfy "the setting
must be readable from `UrlTitle()`". A follow-up PR can add an admin UI
section that calls `SetTransliterateNonLatin` on save and on startup,
without having to re-plumb every `htmltext.UrlTitle` call site through
`context.Context`.
**Default choice — please confirm:** I picked **default-on** because the
existing `topic` behavior is a bug for affected users. If you'd prefer
default-off for strict backward compat on existing installs, flip the
`init()` in `pkg/htmltext/htmltext.go` to `Store(false)` and surface the
toggle as opt-in.
## Why this is safe to ship
- `url_title` is **not** a persisted column. It's not on the `Question`
entity in `internal/entity/question_entity.go`, no migration has ever
added/dropped it, and every call site (`question_service.go`,
`revision_service.go`, `vote_service.go`,
search/report/review/rank/comment services, controllers, repos)
recomputes it from `Title` at response-build time via
`htmltext.UrlTitle(...)`.
- That means the fix is read-only: existing rows light up with correct
slugs on the next request, with no migration and no data rewrite.
- Rollback is just redeploying the prior image; nothing on disk changes.
## Test coverage
`pkg/htmltext/htmltext_test.go`:
- **`TestUrlTitleTable`** — table-driven, one case per affected script
(the full matrix above), plus:
- `empty` → `topic`
- `pure latin unchanged` → byte-identical to pre-fix
- `pure chinese unchanged` → byte-identical to pre-fix (pins existing
pinyin behavior)
- `japanese kanji goes through pinyin path unchanged` → documents the
pre-existing Han-block limitation
- `emoji only falls back to topic` → unchanged
- `long arabic truncates at cutLongTitle boundary` → exercises the
150-byte cap and UTF-8 boundary safety
- **`TestUrlTitleTransliterationToggle`** — with the toggle off,
non-Latin titles collapse to `topic` (pre-fix behavior); with it on,
they transliterate.
- Existing `TestUrlTitle` left untouched.
Test plan for reviewers:
- [ ] `go test ./pkg/htmltext/...` — all pass
- [ ] Visit the live sample URL above and confirm slug is
transliterated, not `topic`
- [ ] Verify Chinese / Latin / emoji-only / empty behavior is
byte-identical to `main` (covered by table tests)
## Out of scope (intentionally)
- No admin UI / site setting plumbing in this PR — see
*Admin-configurable* above. Happy to do the React `Non-Latin Languages
Handling` admin page + `SiteType` + service / controller / migration in
a follow-up if maintainers want it.
- No change to the `"topic"` empty-result fallback.
- No plugin interface for slug generation — mirrored the existing
`convertChinese` pre-step pattern instead.
- No per-language romanization library — this is an explicit non-goal;
see *Transliteration quality* above.
## Issues / discussion
I didn't find an existing upstream issue covering this — happy to be
pointed at one if there is.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: LinkinStars <linkinstar@foxmail.com>
Fix#1524
Root cause
DeepSeek's reasoning models stream reasoning_content alongside content.
Answer ignored it, so follow-up requests failed with 400: The
reasoning_content in the thinking mode must be passed back to the API,
and the thinking text was never shown or saved.
Fix
- Capture reasoning_content from the stream and pass it back to theAPI
on subsequent rounds.
- Persist it with the conversation (new DB column via migrationv2.0.2).
- Render it in the chat UI as a collapsible "Thinking…/Thoughts"panel
above the answer.
Compatibility
Nullable column, omitempty field, UI hides the panel when empty — old
conversations and non-reasoning models behave exactly as before.
Demo
https://github.com/user-attachments/assets/49b1a2a1-9133-4ac2-bbeb-860215a50285
Replace OptionalBool with an explicit require_email_verification value for the login settings save request while keeping legacy read defaults intact.
Use positive RequireEmailVerification naming through the registration flow and inline the site setting mapping.
Add validation, save-path, and registration coverage for the explicit email verification setting.
The ownership check added to AcceptAnswer compared the answer's QuestionID
against the request's QuestionID directly. When short links are enabled,
answerRepo.GetByID re-encodes QuestionID to its short form while the
controller de-shorts req.QuestionID to its long form, so the two encodings
of the same question never matched and every accept returned "Answer do not
found". Normalize both ids via uid.DeShortID before comparing, preserving the
privilege-escalation guard for answers that truly belong to another question.
# fix: avoid `topic` fallback for non-Latin titles via pragmatic ASCII
transliteration
> **Scope update (in response to review):** this PR is intentionally
broader than its original "Arabic-only" framing. The implementation
changes URL slug generation for **every non-Latin, non-CJK script** that
`slugify` previously stripped — see *Scope* below for the explicit list.
The goal is *not* linguistically correct romanization; it is "avoid
collapsing to `/topic` by producing a usable ASCII slug."
## What this PR is (and isn't)
**Goal:** when a question title contains characters outside Basic Latin
/ Latin Extended / CJK Han, generate a URL slug that is a deterministic
ASCII approximation instead of letting `slugify` strip everything and
falling back to the literal `"topic"`.
**Non-goal:** this is *not* a linguistically correct multi-language
romanizer. The output is a machine-acceptable ASCII slug, not what a
native speaker would choose. For example, `こんにちは` → `konnichiha` (not
the more natural `kon'nichiwa`), `ไทย` → `aithy` (not `thai`). Treat the
slug as an opaque, stable, indexable identifier — the
path-after-`/questions/<id>/` is for SEO and shareability, the canonical
reference is always the ID.
## The bug
Pure non-Latin titles previously got stripped by `slugify.Slugify`, hit
the empty-result fallback in `htmltext.UrlTitle`, and collapsed to the
literal slug `"topic"`. On a live multilingual site, every Arabic / Thai
/ Japanese-hiragana / Korean / Hebrew / Cyrillic question ended up at
`/questions/<id>/topic`.
## The fix
`UrlTitle()` gets a `convertNonLatin` pre-step that mirrors the existing
`convertChinese` pre-step pattern, using
`github.com/mozillazg/go-unidecode` (same author as `go-pinyin` already
in the repo, to minimise new-dep friction).
```
UrlTitle(title)
→ convertChinese(title) // pre-existing: Han-block → pinyin
→ convertNonLatin(title) // NEW: detect non-Latin letters → unidecode to ASCII
→ clearEmoji / slugify / url.QueryEscape / cutLongTitle (unchanged)
```
The non-Latin detector skips ASCII, Latin-1 Supplement, Latin
Extended-A/B, and CJK Han. Inputs that hit none of those non-Latin
letter categories short-circuit and return unchanged, so Latin-only and
Chinese-only inputs remain byte-identical (pinned by tests).
## Scope — what scripts are affected
This PR changes behavior for **any** title containing letters in scripts
that `slugify` doesn't handle. Confirmed by tests in
`pkg/htmltext/htmltext_test.go`:
| Script | Example title | Before | After |
| --- | --- | --- | --- |
| Arabic | `كيف حالك` | `topic` | `kyf-hlk` |
| Mixed Latin + Arabic | `مرحبا hello` | `hello` | `mrhb-hello` |
| Thai | `ไทย ไทย` | `topic` | `aithy-aithy` |
| Japanese hiragana | `こんにちは` | `topic` | `konnichiha` |
| Korean | `안녕하세요` | `topic` | `annyeonghaseyo` |
| Hebrew | `שלום עולם` | `topic` | `shlvm-vlm` |
| Cyrillic | `Привет мир` | `topic` | `privet-mir` |
**Unchanged:**
| Case | Behavior |
| --- | --- |
| Pure Latin (`hello world`) | unchanged → `hello-world` |
| Pure Chinese (`这是一个，标题，title`) | unchanged → `zhe-shi-yi-ge-biao-ti`
(pinyin path) |
| Japanese with Han-block kanji (`日本`) | unchanged → `ri-ben` (caught by
pre-existing pinyin path; treated as Chinese reading, not Japanese — a
pre-existing limitation, **not** introduced by this PR) |
| Emoji only (`😂😂😂`) | unchanged → `topic` |
| Empty / whitespace | unchanged → `topic` |
## Transliteration quality — explicit acknowledgement
`go-unidecode` is a generic Unicode → ASCII approximation. It is **not**
a per-language romanization library. Specifically:
- It will pick *one* approximation per codepoint regardless of language
context. `ใ` → `ai` (Thai romanization is `i` or `ai` depending on
standard), `한` → `han`, `語` → `Yu` (Chinese pinyin reading even when
used in Japanese), etc.
- The result is *good enough* to be a stable, URL-safe,
human-recognizable handle, but speakers of the source language will not
consider it "correct."
- It is deterministic, so the same title always produces the same slug —
important since `url_title` is recomputed on every request.
If maintainers prefer to scope this PR more narrowly (e.g. Arabic only,
and reject Thai/Hebrew/Cyrillic/etc.), the detector in
`containsNonLatin` can be tightened to specific Unicode blocks — but
that means the other scripts continue to collapse to `topic`, which is
the bug we're trying to fix. I'd argue the broader fix is preferable to
a piecemeal one, but happy to narrow if you want.
## Live deployment / real-world verification
This patch has been running in production on
**[ask.namasoft.com](https://ask.namasoft.com)** (an Apache Answer
instance we operate) since deployment, built directly from this branch
via `docker compose build`. The site hosts Arabic-language questions, so
the fix exercises the affected code path on every page load.
Sample question URL on the deployed instance:
> `https://ask.namasoft.com/questions/10010000000000115`
The slug in the URL is the transliterated Arabic title rather than
`topic`. No data migration was needed since `url_title` is computed on
every request from `Title` and never persisted (see *Why this is safe to
ship* below).
## Admin-configurable
The transliteration is gated by a package-level `atomic.Bool` (default
**on**, since the current behavior is objectively broken for affected
users):
- `htmltext.SetTransliterateNonLatin(enabled bool)`
- `htmltext.IsTransliterateNonLatinEnabled() bool`
This is deliberately the minimum surface needed to satisfy "the setting
must be readable from `UrlTitle()`". A follow-up PR can add an admin UI
section that calls `SetTransliterateNonLatin` on save and on startup,
without having to re-plumb every `htmltext.UrlTitle` call site through
`context.Context`.
**Default choice — please confirm:** I picked **default-on** because the
existing `topic` behavior is a bug for affected users. If you'd prefer
default-off for strict backward compat on existing installs, flip the
`init()` in `pkg/htmltext/htmltext.go` to `Store(false)` and surface the
toggle as opt-in.
## Why this is safe to ship
- `url_title` is **not** a persisted column. It's not on the `Question`
entity in `internal/entity/question_entity.go`, no migration has ever
added/dropped it, and every call site (`question_service.go`,
`revision_service.go`, `vote_service.go`,
search/report/review/rank/comment services, controllers, repos)
recomputes it from `Title` at response-build time via
`htmltext.UrlTitle(...)`.
- That means the fix is read-only: existing rows light up with correct
slugs on the next request, with no migration and no data rewrite.
- Rollback is just redeploying the prior image; nothing on disk changes.
## Test coverage
`pkg/htmltext/htmltext_test.go`:
- **`TestUrlTitleTable`** — table-driven, one case per affected script
(the full matrix above), plus:
- `empty` → `topic`
- `pure latin unchanged` → byte-identical to pre-fix
- `pure chinese unchanged` → byte-identical to pre-fix (pins existing
pinyin behavior)
- `japanese kanji goes through pinyin path unchanged` → documents the
pre-existing Han-block limitation
- `emoji only falls back to topic` → unchanged
- `long arabic truncates at cutLongTitle boundary` → exercises the
150-byte cap and UTF-8 boundary safety
- **`TestUrlTitleTransliterationToggle`** — with the toggle off,
non-Latin titles collapse to `topic` (pre-fix behavior); with it on,
they transliterate.
- Existing `TestUrlTitle` left untouched.
Test plan for reviewers:
- [ ] `go test ./pkg/htmltext/...` — all pass
- [ ] Visit the live sample URL above and confirm slug is
transliterated, not `topic`
- [ ] Verify Chinese / Latin / emoji-only / empty behavior is
byte-identical to `main` (covered by table tests)
## Out of scope (intentionally)
- No admin UI / site setting plumbing in this PR — see
*Admin-configurable* above. Happy to do the React `Non-Latin Languages
Handling` admin page + `SiteType` + service / controller / migration in
a follow-up if maintainers want it.
- No change to the `"topic"` empty-result fallback.
- No plugin interface for slug generation — mirrored the existing
`convertChinese` pre-step pattern instead.
- No per-language romanization library — this is an explicit non-goal;
see *Transliteration quality* above.
## Issues / discussion
I didn't find an existing upstream issue covering this — happy to be
pointed at one if there is.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: LinkinStars <linkinstar@foxmail.com>
Fix#1524
Root cause
DeepSeek's reasoning models stream reasoning_content alongside content.
Answer ignored it, so follow-up requests failed with 400: The
reasoning_content in the thinking mode must be passed back to the API,
and the thinking text was never shown or saved.
Fix
- Capture reasoning_content from the stream and pass it back to theAPI
on subsequent rounds.
- Persist it with the conversation (new DB column via migrationv2.0.2).
- Render it in the chat UI as a collapsible "Thinking…/Thoughts"panel
above the answer.
Compatibility
Nullable column, omitempty field, UI hides the panel when empty — old
conversations and non-reasoning models behave exactly as before.
Demo
https://github.com/user-attachments/assets/49b1a2a1-9133-4ac2-bbeb-860215a50285
Duansg