Text chat 4: fluxer-emoji crate (#254) #290

Manually merged
charles merged 2 commits from textchat-4-emoji-crate into textchat 2026-11-12 12:51:04 +00:00
Owner

What

New pure crate crates/fluxer-emoji (Text chat 4, spec §4 and §7), depending only on serde/serde_json, no I/O:

  • data.rs: the fork's fluxer_app/src/media/data/emojis.json vendored unchanged (attribution in the README) and parsed once into dataset() -> &'static Dataset (OnceLock). Object order is kept (an order-preserving map visitor, no preserve_order/indexmap), so categories and the 128 shortcuts keep the original's order. Category (8, label(), icon_name() = Phosphor component names from the spec table, plus key()/from_key/ALL), Emoji, SkinVariant, SkinTone (6, plus modifier()/from_modifier for the synced preference), Emoji::with_skin -> Cow<str>, diversity() -> Option<[&str; 5]>. Dataset::{by_name, by_surrogate, in_category, all, shortcuts}; by_name includes the keyword aliases UnicodeEmojis.ts adds (a keyword only one emoji has), by_surrogate ignores FE0F and maps skin-toned spellings to the base emoji.
  • search.rs: search(query, limit) with the six tiers of feat/emoji/state/EmojiSearchIndex.ts (exact name, name prefix, name word boundary incl. plural s/camel case, name contains, keyword prefix, keyword contains), then frecency, then localeCompare of the unique name (approximated: punctuation < digits < letters, accent-insensitive first), then dataset order; query normalised as normalizeEmojiSearchQuery (colons, spaces, +1/?… aliases).
  • twemoji.rs: twemoji_codepoints (FE0F rule incl. the eye-in-speech-bubble exception) and twemoji_url.
  • frecency.rs: Frecency { record, score, top, load, dump, merge } as UsageFrecency.ts (formula, 200 keys, tie-breaks, count cap 100 000, key length 256, clamping).
  • emoticons.rs: emoticon_at(text) -> Option<(name, byte_len)>, first shortcut in dataset order, anchored.
  • jumbo.rs: is_jumbo(tokens) with JumboToken::{Emoji, Text(&str), Other} (≤ 30, whitespace-only text, a text starting with :shortcode: counts as an emoji as in JumboDetector.ts).

How it was tested

  • All the plan's tests, in tests/{data,search,twemoji,frecency}.rs (emoticon and jumbo tests live in data.rs, as the Files list has no separate test files for them), plus names_and_aliases_resolve, tiers_never_go_backwards, match_tiers, query_normalisation, limits_and_frecency, load_sanitises_like_the_original, merge_takes_the_larger_count_and_latest_use, every_dataset_emoji_has_codepoints.
  • The fork has no tests for these modules, so the search goldens (17 queries, top 12) were produced by running the fork's own EmojiSearchIndex.ts + normalizeEmojiSearchQuery under Node 22 (--experimental-strip-types) over the same JSON; the Rust output matches all of them.
  • dataset_loads_fast (#[cfg_attr(debug_assertions, ignore)]) passes with cargo test --release -p fluxer-emoji --test data -- --include-ignored.
  • FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci green locally.

Deviations from the plan

  • Workspace Cargo.toml needed no change: members = ["crates/*", …] already picks the crate up (only Cargo.lock changes).
  • Emoji has public fields (names: Vec<String>, surrogates: String, keywords, category, skins) and same-named accessors (names() -> &[String], surrogates() -> &str, …), so either spelling of the Interfaces block compiles; diversity() is a method.
  • is_jumbo takes impl IntoIterator<Item = JumboToken> (accepts any iterator, as specified, and arrays). Compact mode (never jumbo in the original) is left to the caller.
  • Small additions from the same source files, for Text chat 11/14: Dataset::by_shortcode (name::skin-tone-N), search_with (frecency callback), normalize_query/match_tier/compare_names (to rank guild emoji the same way), Frecency::merge (mergeWireUsageMaps), SkinTone::{modifier, from_modifier}.
  • search with limit == 0 returns everything, as the original's count 0.

Closes #254

🤖 Generated with Claude Code

## What New pure crate `crates/fluxer-emoji` (Text chat 4, spec §4 and §7), depending only on `serde`/`serde_json`, no I/O: - `data.rs`: the fork's `fluxer_app/src/media/data/emojis.json` vendored unchanged (attribution in the README) and parsed once into `dataset() -> &'static Dataset` (`OnceLock`). Object order is kept (an order-preserving map visitor, no `preserve_order`/indexmap), so categories and the 128 shortcuts keep the original's order. `Category` (8, `label()`, `icon_name()` = Phosphor component names from the spec table, plus `key()`/`from_key`/`ALL`), `Emoji`, `SkinVariant`, `SkinTone` (6, plus `modifier()`/`from_modifier` for the synced preference), `Emoji::with_skin -> Cow<str>`, `diversity() -> Option<[&str; 5]>`. `Dataset::{by_name, by_surrogate, in_category, all, shortcuts}`; `by_name` includes the keyword aliases `UnicodeEmojis.ts` adds (a keyword only one emoji has), `by_surrogate` ignores FE0F and maps skin-toned spellings to the base emoji. - `search.rs`: `search(query, limit)` with the six tiers of `feat/emoji/state/EmojiSearchIndex.ts` (exact name, name prefix, name word boundary incl. plural `s`/camel case, name contains, keyword prefix, keyword contains), then frecency, then `localeCompare` of the unique name (approximated: punctuation < digits < letters, accent-insensitive first), then dataset order; query normalised as `normalizeEmojiSearchQuery` (colons, spaces, `+1`/`?`… aliases). - `twemoji.rs`: `twemoji_codepoints` (FE0F rule incl. the eye-in-speech-bubble exception) and `twemoji_url`. - `frecency.rs`: `Frecency { record, score, top, load, dump, merge }` as `UsageFrecency.ts` (formula, 200 keys, tie-breaks, count cap 100 000, key length 256, clamping). - `emoticons.rs`: `emoticon_at(text) -> Option<(name, byte_len)>`, first shortcut in dataset order, anchored. - `jumbo.rs`: `is_jumbo(tokens)` with `JumboToken::{Emoji, Text(&str), Other}` (≤ 30, whitespace-only text, a text starting with `:shortcode:` counts as an emoji as in `JumboDetector.ts`). ## How it was tested - All the plan's tests, in `tests/{data,search,twemoji,frecency}.rs` (emoticon and jumbo tests live in `data.rs`, as the Files list has no separate test files for them), plus `names_and_aliases_resolve`, `tiers_never_go_backwards`, `match_tiers`, `query_normalisation`, `limits_and_frecency`, `load_sanitises_like_the_original`, `merge_takes_the_larger_count_and_latest_use`, `every_dataset_emoji_has_codepoints`. - The fork has no tests for these modules, so the search goldens (17 queries, top 12) were produced by running the fork's own `EmojiSearchIndex.ts` + `normalizeEmojiSearchQuery` under Node 22 (`--experimental-strip-types`) over the same JSON; the Rust output matches all of them. - `dataset_loads_fast` (`#[cfg_attr(debug_assertions, ignore)]`) passes with `cargo test --release -p fluxer-emoji --test data -- --include-ignored`. - `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` green locally. ## Deviations from the plan - Workspace `Cargo.toml` needed no change: `members = ["crates/*", …]` already picks the crate up (only `Cargo.lock` changes). - `Emoji` has public fields (`names: Vec<String>`, `surrogates: String`, `keywords`, `category`, `skins`) and same-named accessors (`names() -> &[String]`, `surrogates() -> &str`, …), so either spelling of the Interfaces block compiles; `diversity()` is a method. - `is_jumbo` takes `impl IntoIterator<Item = JumboToken>` (accepts any iterator, as specified, and arrays). Compact mode (never jumbo in the original) is left to the caller. - Small additions from the same source files, for Text chat 11/14: `Dataset::by_shortcode` (`name::skin-tone-N`), `search_with` (frecency callback), `normalize_query`/`match_tier`/`compare_names` (to rank guild emoji the same way), `Frecency::merge` (`mergeWireUsageMaps`), `SkinTone::{modifier, from_modifier}`. - `search` with `limit == 0` returns everything, as the original's `count` 0. Closes #254 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Refs #254

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Author
Owner

Independent review: Text chat 4 (fluxer-emoji), commit 2d87335

Scope: git diff 0283642..origin/textchat-4-emoji-crate (crate crates/fluxer-emoji + Cargo.lock only).

What I checked

  • Plan tests: all 10 named tests are present (every_surrogate_maps_back_to_its_first_name, categories_in_original_order, skin_tone_variants, search_ranks_exact_then_prefix_then_keyword, twemoji_codepoints_cases with the six spec cases ❤️ 👨‍👩‍👧 🏳️‍🌈 👍🏽 🇫🇷 ©️, frecency_matches_formula_and_tie_breaks, frecency_keeps_200_keys, emoticons_match_original_list, jumbo_rule_30_and_text, dataset_loads_fast ignored in debug). Labels and icons match spec §3.4 (Smiley, Leaf, BowlFood, GameController, Bicycle, Magnet, Heart, Flag); formula, 200 keys and the 30-emoji jumbo limit match.
  • Interfaces: Category + label()/icon_name(), Emoji fields/accessors (names, surrogates, keywords, category, diversity() -> Option<[&str; 5]>), dataset() (OnceLock), Dataset::{by_name, by_surrogate, in_category, all, shortcuts}, SkinTone (6), with_skin -> Cow<str>, search(query, limit), twemoji_codepoints, twemoji_url ({cdn}/emoji/{cp}.svg?v=2), Frecency::{record, score, top} + load/dump over BTreeMap<String, (u32, i64)>, emoticon_at -> Option<(&str, usize)>, is_jumbo (takes IntoIterator, so an Iterator works). All present.
  • Against the fork: emojis.json is byte-identical to fluxer_app/src/media/data/emojis.json. Match tiers and boundary rule follow EmojiSearchIndex.ts, the query normaliser follows normalizeEmojiSearchQuery, codepoints follow EmojiCodepointUtils.ts (including the eye-in-speech-bubble exception), frecency sanitising and tie-breaks follow UsageFrecency.ts (compareStrings = UTF-16 order), and keyword aliases follow buildEmojiIndex. The emoticon order matches the JS Object.keys order: no integer-like keys, no duplicates, and no shortcut is a prefix of a later one.
  • compare_names (the implementer's flagged approximation): I sorted all 1940 unique names with Node 22 localeCompare and with compare_names. The two orders are identical.
  • Invariants: depends only on serde/serde_json; no I/O (only include_str!); no paths, URLs or secrets in code. On server-sourced input (frecency maps, shortcodes, surrogates, queries) I found no panic path: clamp bounds are safe, and indexing is guarded.
  • CI: FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci is green in a clean review worktree. cargo test --release -p fluxer-emoji -- --include-ignored also passes (dataset_loads_fast < 50 ms).

Findings

  1. (minor) search("", n) returns the first n emoji. The original's EmojiSearchIndex.search returns every emoji for an empty query and ignores count. This is harmless for the picker (it does not search with an empty query), but it is a small divergence. Either document it or drop the cap.
  2. (minor) by_surrogate maps every skin-toned spelling to the base emoji, including the 20 non-uniform multi-person combinations, for which the original's surrogateToName gives the variant's own name. This is documented. Task 8 needs to know about it: when rendering, it must build the Twemoji URL from the source text, not from emoji.surrogates, or toned emoji will draw untoned.
  3. (minor) Frecency::load(map, now_ms) sanitises like sanitizeUsageMap: it clamps last_used to now, drops entries with count 0 or an invalid key, and prunes to 200. So load then dump is not the identity. This matches the original, but Task 14 must write frecency back only through the picker-field edit of the Task 3 codec, so the byte-for-byte round-trip of the other fields still holds.

No blocking findings.

Verdict: approve (0 blocking, 3 minor; ci green locally).

## Independent review: Text chat 4 (fluxer-emoji), commit 2d87335 Scope: `git diff 0283642..origin/textchat-4-emoji-crate` (crate `crates/fluxer-emoji` + Cargo.lock only). **What I checked** - Plan tests: all 10 named tests are present (`every_surrogate_maps_back_to_its_first_name`, `categories_in_original_order`, `skin_tone_variants`, `search_ranks_exact_then_prefix_then_keyword`, `twemoji_codepoints_cases` with the six spec cases ❤️ 👨‍👩‍👧 🏳️‍🌈 👍🏽 🇫🇷 ©️, `frecency_matches_formula_and_tie_breaks`, `frecency_keeps_200_keys`, `emoticons_match_original_list`, `jumbo_rule_30_and_text`, `dataset_loads_fast` ignored in debug). Labels and icons match spec §3.4 (Smiley, Leaf, BowlFood, GameController, Bicycle, Magnet, Heart, Flag); formula, 200 keys and the 30-emoji jumbo limit match. - Interfaces: `Category` + `label()`/`icon_name()`, `Emoji` fields/accessors (`names`, `surrogates`, `keywords`, `category`, `diversity() -> Option<[&str; 5]>`), `dataset()` (OnceLock), `Dataset::{by_name, by_surrogate, in_category, all, shortcuts}`, `SkinTone` (6), `with_skin -> Cow<str>`, `search(query, limit)`, `twemoji_codepoints`, `twemoji_url` (`{cdn}/emoji/{cp}.svg?v=2`), `Frecency::{record, score, top}` + load/dump over `BTreeMap<String, (u32, i64)>`, `emoticon_at -> Option<(&str, usize)>`, `is_jumbo` (takes `IntoIterator`, so an `Iterator` works). All present. - Against the fork: `emojis.json` is byte-identical to `fluxer_app/src/media/data/emojis.json`. Match tiers and boundary rule follow `EmojiSearchIndex.ts`, the query normaliser follows `normalizeEmojiSearchQuery`, codepoints follow `EmojiCodepointUtils.ts` (including the eye-in-speech-bubble exception), frecency sanitising and tie-breaks follow `UsageFrecency.ts` (compareStrings = UTF-16 order), and keyword aliases follow `buildEmojiIndex`. The emoticon order matches the JS `Object.keys` order: no integer-like keys, no duplicates, and no shortcut is a prefix of a later one. - `compare_names` (the implementer's flagged approximation): I sorted all 1940 unique names with Node 22 `localeCompare` and with `compare_names`. The two orders are **identical**. - Invariants: depends only on serde/serde_json; no I/O (only `include_str!`); no paths, URLs or secrets in code. On server-sourced input (frecency maps, shortcodes, surrogates, queries) I found no panic path: clamp bounds are safe, and indexing is guarded. - CI: `FLUXER_SCREENSHOT_TESTS=1 cargo xtask ci` is **green** in a clean review worktree. `cargo test --release -p fluxer-emoji -- --include-ignored` also passes (dataset_loads_fast < 50 ms). **Findings** 1. (minor) `search("", n)` returns the first `n` emoji. The original's `EmojiSearchIndex.search` returns every emoji for an empty query and ignores `count`. This is harmless for the picker (it does not search with an empty query), but it is a small divergence. Either document it or drop the cap. 2. (minor) `by_surrogate` maps every skin-toned spelling to the **base** emoji, including the 20 non-uniform multi-person combinations, for which the original's `surrogateToName` gives the variant's own name. This is documented. Task 8 needs to know about it: when rendering, it must build the Twemoji URL from the source text, not from `emoji.surrogates`, or toned emoji will draw untoned. 3. (minor) `Frecency::load(map, now_ms)` sanitises like `sanitizeUsageMap`: it clamps `last_used` to `now`, drops entries with count 0 or an invalid key, and prunes to 200. So `load` then `dump` is not the identity. This matches the original, but Task 14 must write frecency back only through the picker-field edit of the Task 3 codec, so the byte-for-byte round-trip of the other fields still holds. No blocking findings. **Verdict: approve** (0 blocking, 3 minor; ci green locally).
charles manually merged commit 35167503c6 into textchat 2026-11-12 12:51:04 +00:00
Sign in to join this conversation.
No description provided.