tds3p-rsvp-reader/docs/history/PLAN.md

Historical text coloring implementation plan

This proposal predates the current v3 sidecar and dialogue coloring implementation. Its entity format, palette, and rollout order are not the current design. See the root README for implemented behavior and TODO.md for remaining work.

Goal and order

Validate color rendering on the T-Display S3 Pro before changing EPUB attribution. First add firmware support for sparse v3 entity metadata and a card-ready lorem ipsum book that repeats the full palette. Flash the board and inspect the colors, gray, white, word transitions, and reading speed. Only then build the converter's dialogue and thought attribution.

The revised palette has nine hues: 0, 30, 60, 120, 180, 225, 270, 300, and 330 degrees, all at HSL saturation 100% and lightness 75%. HSL (0, 0%, 75%) is the gray for Other and uncolored entities. Ordinary words remain white. Hue 360 is equivalent to 0, so the map uses 0.

Phase 1: firmware and hardware book

1. Accept v3 RSVP sidecars while preserving v1/v2 behavior. V3 keeps the existing magic, source fingerprint, and record count, then adds a 64-bit entity-catalog fingerprint. Each sparse record retains its word-index delta, style flags, and timing modifier. The high bit of the style byte means an entity ID follows as an unsigned varint. The entity ID may be 0, which is the real category Other. An absent high bit means no entity. A word with no style, timing, or entity assignment has no sidecar record.
2. Load an editable ENTITY_COLORS.tsv once per book. It maps entity IDs to hues, with an entity-catalog fingerprint to reject a stale map. ID 0 always uses gray; named entities with no mapped hue also use gray. A missing or invalid map leaves assigned words gray. Multiple entities may share a hue; the map has a 64 KiB file-size limit but no entity-count limit. V1/v2 books remain readable.
3. Add a saved Colors on/off setting and a sixth menu item. Off is the default and renders all words white as before. On colors only entity-tagged reading words; chapter cards, menus, and ordinary words remain white. Convert hues to RGB565 when loading the book and keep the palette in memory, with no SD reads during word advances.
4. Include a card-ready Color Cycle book containing only repeated lorem ipsum words. In each 55-word cycle, each of the nine hues colors five consecutive words, followed by five words with entity ID 0 (gray) and five with no entity annotation (white). Repeat the cycle many times so playback and manual navigation expose rendering artifacts.
5. Build the firmware and verify the fixture's word order, v3 fingerprint, map fingerprint, and color annotations. Hardware acceptance requires the user to observe all eleven five-word color blocks across repeated cycles, clean transitions, and the persisted toggle after a restart.

Phase 2: attribution rules and statistics

1. Preserve quote boundaries, EPUB style cues, and final word indexes through normalization and character substitution. Identify conventional paired quotation marks and only clearly marked or context-supported thought spans; generic italics alone do not imply thought.
2. Attribute explicit named speech/thought tags as clear. Use surrounding context and a bounded two-speaker exchange for reasonable guesses, including multi-paragraph continuations. Explicit pronoun-only tags such as he, she, and it map to Other. Unresolved dialogue/thought also maps to Other; text that is not dialogue/thought remains unassigned.
3. Process the example book iteratively and report aggregate word and span counts for clear, guessed, and unattributable cases. Refine the rules until the user likes that distribution. Do not present passages, per-span review, or a manual speaker/alias correction workflow in the terminal UI.
4. Detect canonical character names and safe aliases automatically. Color name mentions outside dialogue by their entity; inside dialogue, the speaker's color takes precedence. Ambiguous name matches remain unassigned.

Phase 3: terminal color assignment and converter output

After attribution, list entities by descending dialogue word count in the terminal, without excerpts. Let the user assign any of the nine hues to each entity; the same hue may be assigned to multiple entities. The user may stop at any point, leaving remaining named entities gray. Other remains gray. Display ANSI true-color swatches when supported, with numeric hue labels as the fallback. Write the editable per-book map and v3 sidecars, keeping metadata sparse and leaving the source text unchanged. Update the README and cover token alignment, attribution statistics, sidecar compatibility, map failures, size limits, and on-device performance.