j2
/
tds3p-rsvp-reader
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add dialogue detection and coloring, some tests for attribution but little success

This commit is contained in:
j2 2026-10-11 23:16:22 -05:00
parent ba65e49856
commit e06b886c28
29 changed files with 4075 additions and 156 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
.github/workflows/ci.yml vendored
View File

@ -7,27 +7,20 @@ on:
jobs:
converter:
runs-on: ubuntu-latest
defaults:
run:
working-directory: book-processing
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python -m pip install -r requirements.txt
- run: python -m unittest discover -v
- run: python epub_to_chapters.py examples/rsvp-timing-sample.epub /tmp/timing-sample
python-version: '3.11'
- run: make test
- run: make convert
firmware:
runs-on: ubuntu-latest
defaults:
run:
working-directory: firmware
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
python-version: '3.11'
- run: python -m pip install platformio
- run: pio run
- run: make build

2
.gitignore vendored
View File

@ -1,5 +1,6 @@
# PlatformIO build output and local editor settings.
.pio/
/.platformio/
.vscode/
*.bin
*.elf
@ -16,6 +17,7 @@ htmlcov/
# Generated reader content and local source books.
chapters/
/Wind and Truth/
*.epub
!book-processing/examples/rsvp-timing-sample.epub

2
Makefile
View File

@ -1,5 +1,7 @@
PYTHON ?= python3
PIO ?= pio
PLATFORMIO_CORE_DIR ?= $(CURDIR)/.platformio
export PLATFORMIO_CORE_DIR
UPLOAD_PORT ?=
VENV := .venv
DEPS_STAMP := $(VENV)/.requirements-installed

92
README.md
View File

@ -1,13 +1,14 @@
# T-Display S3 Pro RSVP Reader
An offline-first RSVP reader for the LilyGO T-Display S3 Pro. It renders one
white word on a black, landscape display and never starts Wi-Fi or Bluetooth.
An offline RSVP reader for the LilyGO T-Display S3 Pro. It renders one word at
a time on a black, landscape display and never starts Wi-Fi or Bluetooth.
Detected dialogue appears grey when Colors is on; other words appear white.
## What you need
- A LilyGO T-Display S3 Pro and a **USB-C data cable**. A charge-only cable
cannot upload firmware.
- A computer with internet access for the first build, Python 3, and a terminal.
- A computer with internet access for the first build, Python 3.11, and a terminal.
- A microSD card formatted as FAT32. The reader uses this as its book library.
- This repository, cloned or downloaded to your computer.
@ -67,13 +68,16 @@ operating systems: <https://docs.platformio.org/en/latest/core/installation/meth
1. Format the microSD card as **FAT32**.
2. Create a directory named `books` at the card's root.
3. Create one folder per book inside `books`. Each book folder needs a
`TOC.txt`, UTF-8 text files for its sections, and matching `.rsvp` sidecars
generated by the converter.
`TOC.txt` and a `content` folder containing UTF-8 section text files and
matching `.rsvp` sidecars generated by the converter.
`TOC.txt` is the reader's source of truth for section order, membership, and
title-card text. Each non-empty line names a section and its file is that name
plus `.txt`. The reader reads and caches this file when it needs navigation
order; it opens only the selected section file.
title-card text. Each non-empty line names a section and its file is
`content/<section>.txt`. The reader reads and caches this file for navigation;
it opens only the selected section file.
Chapter titles such as `TOC` and `CONVERSION_REPORT` are valid because their
files live in `content/`. For an older card, move each section's `.txt` and
`.rsvp` files into `content/`, leaving `TOC.txt` and book metadata at the root.
The resulting card must look like this:
@ -82,6 +86,8 @@ microSD root/
└── books/
└── Example Book/
├── TOC.txt
├── CONVERSION_REPORT.txt
└── content/
├── Dedication.txt
├── Dedication.rsvp
├── Chapter 1.txt
@ -98,22 +104,24 @@ Chapter 1
Chapter 2
```
Generate this layout from an EPUB with the included converter:
From the repository root, install the Python dependencies and convert an EPUB:
```bash
./.venv/bin/python -m pip install -r book-processing/requirements.txt
make deps
./.venv/bin/python book-processing/epub_to_chapters.py book.epub "Example Book"
```
For a quick end-to-end check, the repository includes a two-chapter sample
with EPUB styles, punctuation, and repeated invented names:
For a quick end-to-end check, convert the included two-chapter sample:
```bash
./.venv/bin/python book-processing/epub_to_chapters.py book-processing/examples/rsvp-timing-sample.epub "Timing Sample"
make convert
```
The output directory must be empty unless `--overwrite` is supplied. Copy the
resulting `Example Book` directory into `/books` on the card. A section is
The sample command prints its temporary output directory. For your own book,
the output directory must be empty unless `--overwrite` is supplied. On
overwrite, the converter replaces generated chapter files in `content/` and
the root TOC and conversion report while preserving other root metadata. Copy
the resulting `Example Book` directory into `/books` on the card. A section is
limited to 150,000 words; its source text and `.rsvp` sidecar may each be up
to 1 MiB. The converter writes
`CONVERSION_REPORT.txt` in the book directory. If unsupported non-ASCII
@ -125,8 +133,12 @@ chosen substitutions. The included display font is best for printable ASCII.
The converter also writes a compact `<section>.rsvp` file alongside every
section. Its versioned, delta-encoded records preserve italic and bold EPUB
markup—including semantic tags, CSS classes, and inline CSS—and reading-timing
metadata. The fingerprint covers the source `.txt` bytes, before the device
markup—including semantic tags, CSS classes, and inline CSS—reading-timing
metadata, and a word color flag. Quoted dialogue and wholly italic or uppercase
punctuated clauses receive the `Other` flag; speaker attribution is not used
during conversion. With coloring enabled, the device shows those words in grey
and other words in white. The reader menu has a persisted **Colors: On/Off**
setting. The fingerprint covers the source `.txt` bytes, before the device
applies display-only punctuation normalization. During conversion, the
desktop-only English vocabulary classifier marks each unfamiliar word. It
combines a dictionary with word-frequency data and recognizes normal English
@ -141,25 +153,38 @@ bold records select matching reader-font variants; the device never performs
dictionary or SD-card lookups while advancing words.
The converter requires the desktop-only `english-words` and `wordfreq`
packages listed in `book-processing/requirements.txt`; the reader does not need either
dependency. Reconvert a book to create v2 sidecars with unfamiliar-word
timing. Existing v1 style-only sidecars remain readable but do not contain
timing metadata.
packages listed in `book-processing/requirements.txt`; the reader does not
need either dependency. Reconvert a book to create v3 sidecars with dialogue coloring.
Existing v1/v2 sidecars remain readable; v1 has no timing metadata, and neither
older version has word color flags.
Speaker attribution remains an evaluation tool and is not part of EPUB
conversion. `book-processing/attribution.py` contains experimental rules,
including Maverick Coref for pronouns in speech tags. The default `make test`
uses a stub model and the labeled Prologue and Chapter 1 fixtures. To run the
optional evaluation with the downloaded model, use:
```bash
RUN_MAVERICK_EVAL=1 make test
```
That first run downloads the LitBank model and tokenizer data. Its results do
not change the generated book or sidecars.
Eject the card safely and insert it in the board before powering the board.
## 3. Build the firmware
Open a terminal in the `firmware` directory:
From the repository root, build through the Makefile:
```bash
cd /path/to/tds3p-rsvp-reader/firmware
pio run
make build
```
On the first run, PlatformIO downloads the pinned ESP32 platform, compiler,
and the TFT_eSPI, SensorLib, and XPowersLib dependencies. A successful build
ends with `SUCCESS` and creates the firmware under `.pio/build/rsvp_reader/`.
and the TFT_eSPI, SensorLib, and XPowersLib dependencies into the repository's
`.platformio/` cache. A successful build ends with `SUCCESS` and creates the
firmware under `firmware/.pio/build/rsvp_reader/`.
The project targets the T-Display S3 Pro's 16 MB flash and 8 MB OPI PSRAM. It
defaults to the v1.1 16-step backlight driver. If your board is the original
@ -188,13 +213,13 @@ on macOS, and `COM3` on Windows. Note the port belonging to the board.
Let PlatformIO select the port automatically:
```bash
pio run -t upload
make flash
```
If it selects the wrong port or does not find one, supply it explicitly:
```bash
pio run -t upload --upload-port /dev/ttyACM0
make flash UPLOAD_PORT=/dev/ttyACM0
```
Replace `/dev/ttyACM0` with the port from the previous step. On Windows, for
@ -243,8 +268,9 @@ The reader saves the selected section, paragraph-level position, WPM,
brightness, and font settings in the ESP32's nonvolatile storage. On the next
boot or wake it opens the same section's title card and remains paused;
pressing BOOT begins at the saved paragraph. It opens the
saved section directly, then falls back to the first TOC-listed section if the
card has changed.
saved section directly, maps a saved path from the old flat book layout to
`content/` when possible, then falls back to the first TOC-listed section if
the card has changed.
When continuous reading reaches the end of a section, it opens the next
section's title card and pauses. Press BOOT to begin that
@ -261,17 +287,17 @@ cloud account, or synchronization feature.
- **Permission denied on Linux:** complete the `dialout` group step and log out
and back in.
- **`NO BOOKS`:** reformat the card FAT32, create `/books/<book>`, and add a
`TOC.txt` plus the listed `.txt` section files. Confirm the card is fully
`TOC.txt` plus the listed `.txt` section files under `content/`. Confirm the card is fully
seated. Dot-prefixed folders such as `.Trash` are ignored.
- **Black display after boot:** press GPIO12 to wake it if it has slept; otherwise
reflash and check whether the v1.0 backlight build flag is needed.
- **Build fails before compilation:** rerun `pio run` with internet access so
- **Build fails before compilation:** rerun `make build` with internet access so
PlatformIO can download the required packages.
## Limits of the first MVP
- It reads sections up to 150,000 words and 1 MiB from
`/books/<book>/<section>.txt`, in the order declared by that book's
`/books/<book>/content/<section>.txt`, in the order declared by that book's
`TOC.txt`.
- The built-in display font is best for printable ASCII text. UTF-8 text files
are accepted, but accented/non-Latin characters need a Unicode font addition

26
TODO.md
View File

@ -1,4 +1,10 @@
# TODO
# Project status and next steps
The converter writes TOC and book metadata at the book root, with chapter text
and v3 RSVP sidecars in `content/`. It detects dialogue and marks it `Other`;
the firmware displays those words in grey when Colors is on. Named speaker
attribution and Maverick Coref remain outside the conversion flow and are used
only in optional evaluation.
## Reading profiles
@ -9,24 +15,18 @@ profile that preserves the current cadence.
Replace fixed 2×/3× sidecar word-delay values with categorical timing tags.
Map each category to a configurable delay in reader settings, so changing a
delay does not require regenerating sidecars. Version the format change and
keep existing v1/v2 sidecars readable.
keep existing v1/v2/v3 sidecars readable.
## Dialogue and character colors
Add optional color rendering for dialogue and character names. The converter
should detect conventional quotation marks and styled thought/dialogue spans,
then attribute only high-confidence spans to a canonical entity. Store the
entity identity, never a color, in a compact v3 RSVP sidecar; load a separate,
editable per-book entity map that assigns each entity a hue from 30 through 330
Attribute only high-confidence spans to a canonical entity and load a separate, editable
per-book entity map that assigns each named entity a hue from 30 through 330
in 30-degree HSL steps at 100% saturation and 75% lightness. Include aliases
so descriptions such as "mysterious man" can later resolve to a named entity,
and report unresolved or ambiguous attribution for review rather than guessing.
Keep the sidecar sparse: unassigned words must have no sidecar record at all,
not a record with an entity ID of zero or another null sentinel. Extend a
record only when it already exists for style/timing metadata or when an entity
assignment requires one. Add a persisted reader setting to toggle coloring
without requiring reconversion.
Keep later entity assignments sparse: unassigned words have no sidecar record
unless style or timing metadata requires one.
## Add anti-aliased reader fonts
@ -41,7 +41,7 @@ previews.
3.5, and 4.
2. Place the files in a dedicated SD-card directory, for example
`/fonts/<style>-<size>.vlw`; document these required files alongside the
existing `/books/<book>/<chapter>.txt` layout.
existing `/books/<book>/content/<chapter>.txt` layout.
3. Change `Board` to load the selected smooth font from SD when a font setting
is applied, and fall back to the current bitmap font if the file is absent
or cannot be loaded.

435
book-processing/attribution.py Normal file
View File

@ -0,0 +1,435 @@
"""Dialogue detection and speaker attribution rules for chapter evaluation."""
import re
from collections import defaultdict
from dataclasses import dataclass
from functools import lru_cache
from typing import Any
WORD = re.compile(r"\S+")
CLAUSE_PUNCTUATION = re.compile(r"[.!?;,:—–]")
PARAGRAPH_BREAK = re.compile(r"\r?\n[ \t]*\r?\n+")
PAIRED_SINGLE_QUOTES = re.compile(r"(?<!\w)'[^'\n]+'(?!\w)")
REPORTING_VERB = (
r"(?:said|asked|replied|whispered|muttered|shouted|demanded|"
r"admitted|grumbled|called|cried|answered|warned|added)"
)
SPEECH_TAG = re.compile(
r"\b(he|she|they|it)\s+(?:(?:softly|quietly)\s+)?"
+ REPORTING_VERB + r"\b",
re.IGNORECASE,
)
PRONOUNS = frozenset(("he", "she", "they", "it"))
MAVERICK_TOKEN_LIMIT = 500
@dataclass(frozen=True)
class AttributionStats:
total_paragraphs: int
labeled_paragraphs: int
unlabeled_paragraphs: int
expected_dialogue: int
expected_non_dialogue: int
correct_assignments: int
incorrect_assignments: int
missed_dialogue: int
extra_dialogue: int
correct_non_dialogue: int
predicted_on_unlabeled: int
def identify_dialogue(paragraph, word_start, italic_word_indexes, paragraph_index=None):
"""Detect quoted text or a fully italic or uppercase punctuated clause."""
if '"' in paragraph:
return True
return bool(_styled_dialogue_word_indexes(
paragraph, word_start, italic_word_indexes
))
def _styled_dialogue_word_indexes(paragraph, word_start, italic_word_indexes):
"""Find words in wholly italic or uppercase punctuated clauses."""
words = list(WORD.finditer(paragraph))
clauses = {}
for word_number, match in enumerate(words):
letters = "".join(character for character in match.group()
if character.isalpha())
clause_number = len(CLAUSE_PUNCTUATION.findall(paragraph[:match.start()]))
clause = clauses.setdefault(
clause_number,
{"italic": True, "uppercase": True, "last_start": match.start(),
"first_start": match.start(), "word_indexes": []},
)
clause["italic"] &= word_start + word_number in italic_word_indexes
clause["uppercase"] &= bool(letters) and letters.isupper()
clause["last_start"] = match.start()
clause["word_indexes"].append(word_start + word_number)
ordered = list(clauses.values())
dialogue_indexes = set()
for index, clause in enumerate(ordered):
boundary_end = (ordered[index + 1]["first_start"]
if index + 1 < len(ordered) else len(paragraph))
punctuated = CLAUSE_PUNCTUATION.search(
paragraph, clause["last_start"], boundary_end
) is not None
if punctuated and (clause["italic"] or clause["uppercase"]):
dialogue_indexes.update(clause["word_indexes"])
return dialogue_indexes
def dialogue_word_indexes(text, style_records):
"""Find quoted and styled dialogue words without resolving speakers."""
italic_indexes = {
index for index, flags, *_ in style_records if flags & 0x01
}
dialogue_indexes = set()
word_start = 0
in_double_quote = False
for raw_paragraph in PARAGRAPH_BREAK.split(text):
paragraph = raw_paragraph.strip()
if not paragraph:
continue
words = list(WORD.finditer(paragraph))
dialogue_indexes.update(_styled_dialogue_word_indexes(
paragraph, word_start, italic_indexes
))
single_quotes = list(PAIRED_SINGLE_QUOTES.finditer(paragraph))
continuation_quote = in_double_quote and paragraph.startswith('"')
for word_number, match in enumerate(words):
quoted_word = False
for position in range(match.start(), match.end()):
character = paragraph[position]
if character == '"':
quoted_word = True
if continuation_quote and position == 0:
continuation_quote = False
else:
in_double_quote = not in_double_quote
elif in_double_quote:
quoted_word = True
if quoted_word or any(
match.start() < quote.end() and match.end() > quote.start()
for quote in single_quotes
):
dialogue_indexes.add(word_start + word_number)
word_start += len(words)
return dialogue_indexes
def speech_tag_pronouns(paragraph):
"""Find reporting-clause pronouns outside double-quoted speech."""
return [match.start(1) for match in SPEECH_TAG.finditer(paragraph)
if paragraph[:match.start(1)].count('"') % 2 == 0]
def named_speech_tag_speaker(paragraph, speaker_names):
"""Return a unique named speaker followed by a reporting verb outside quotes."""
outside_quotes = []
quoted = False
for character in paragraph:
if character == '"':
quoted = not quoted
outside_quotes.append(" ")
else:
outside_quotes.append(" " if quoted else character)
text = "".join(outside_quotes)
matches = set()
for name in speaker_names:
if name == "Other":
continue
pattern = re.compile(
r"(?<!\w)" + re.escape(name) + r"(?!\w)"
r"\s+(?:(?:softly|quietly)\s+)?" + REPORTING_VERB + r"\b",
re.IGNORECASE,
)
if pattern.search(text):
matches.add(name)
return next(iter(matches)) if len(matches) == 1 else None
@lru_cache(maxsize=2)
def load_maverick(device="cpu"):
"""Load the literary model with only its checkpoint metadata allowlisted."""
try:
from maverick import Maverick
from omegaconf.base import ContainerMetadata, Metadata
from omegaconf.dictconfig import DictConfig
from omegaconf.nodes import AnyNode
from torch.serialization import safe_globals
except ImportError as exc:
raise RuntimeError(
"Maverick attribution requires maverick-coref; install "
"book-processing/requirements.txt in the project venv"
) from exc
# PyTorch 2.6 loads checkpoints in weights-only mode. These are the
# non-tensor types in the published LitBank checkpoint's Hydra metadata.
with safe_globals([Any, dict, AnyNode, defaultdict, Metadata,
ContainerMetadata, DictConfig]):
return Maverick(hf_name_or_path="sapienzanlp/maverick-mes-litbank",
device=device)
def _align_tokens(text, tokens):
"""Align Maverick tokens against the input, preserving actual whitespace.
Maverick's plain-text char offsets assume one character between sentences,
which is not true for our blank-line-delimited chapters.
"""
spans = []
cursor = 0
for token in tokens:
start = text.find(token, cursor)
if start < 0:
raise ValueError(f"Cannot align Maverick token {token!r} after {cursor}")
end = start + len(token)
spans.append((start, end))
cursor = end
return spans
def _cluster_pronouns(text, result, speaker_names):
"""Map pronoun character starts to a name only for unambiguous clusters."""
token_spans = _align_tokens(text, result["tokens"])
names = tuple(name for name in speaker_names if name != "Other")
mappings = {}
for cluster in result["clusters_token_offsets"]:
mentions = [text[token_spans[start][0]:token_spans[end][1]]
for start, end in cluster]
matches = {
name for name in names for mention in mentions
if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)",
mention, re.IGNORECASE)
}
if len(matches) != 1:
continue
name = next(iter(matches))
for start, end in cluster:
mention = text[token_spans[start][0]:token_spans[end][1]]
if mention.casefold() in PRONOUNS:
mappings[token_spans[start][0]] = name
return mappings
def _maverick_token_count(model, text):
"""Count the input tokens Maverick will send to its encoder."""
if hasattr(model, "token_count"):
return model.token_count(text)
tokens, _, _, _ = model.preprocess(text)
input_tokens = ["[SPEAKER_START]", "-", "[SPEAKER_END]", *tokens]
return len(model.tokenizer(input_tokens, add_special_tokens=True,
is_split_into_words=True)["input_ids"])
def _window_start(text, sentence_start, sentence_end, model):
"""Keep the target sentence and the most preceding words within budget."""
if _maverick_token_count(model, text[sentence_start:sentence_end]) > MAVERICK_TOKEN_LIMIT:
raise ValueError("A reporting-clause sentence exceeds Maverick's token budget")
starts = sorted({0, sentence_start, *(match.start() for match in
WORD.finditer(text[:sentence_start]))})
left, right = 0, len(starts) - 1
while left < right:
middle = (left + right) // 2
if _maverick_token_count(model, text[starts[middle]:sentence_end]) <= MAVERICK_TOKEN_LIMIT:
right = middle
else:
left = middle + 1
while _maverick_token_count(model, text[starts[left]:sentence_end]) > MAVERICK_TOKEN_LIMIT:
left += 1
while left and _maverick_token_count(model, text[starts[left - 1]:sentence_end]) <= MAVERICK_TOKEN_LIMIT:
left -= 1
return starts[left]
def _maverick_resolutions(paragraphs, speaker_names, model, target_pronouns):
"""Resolve each tag in its sentence and revisit earlier tags in later runs."""
from nltk.data import load as load_nltk_resource
sentence_tokenizer = load_nltk_resource("tokenizers/punkt/english.pickle")
text = "\n\n".join(paragraph["text"] for paragraph in paragraphs)
offset = 0
runs = []
tag_positions = {}
for paragraph in paragraphs:
index = paragraph["index"]
tags = target_pronouns.get(index, ())
if tags:
tag_positions[index] = tuple(offset + tag for tag in tags)
spans = list(sentence_tokenizer.span_tokenize(paragraph["text"]))
for start, end in spans:
if any(start <= tag < end for tag in tags):
runs.append((index, offset + start, offset + end))
offset += len(paragraph["text"]) + 2
initial = {}
later = {}
for current_index, sentence_start, sentence_end in runs:
start = _window_start(text, sentence_start, sentence_end, model)
mappings = _cluster_pronouns(
text[start:sentence_end], model.predict(text[start:sentence_end]),
speaker_names,
)
for index, positions in tag_positions.items():
if index in initial or index in later:
continue
if not all(start <= position < sentence_end for position in positions):
continue
names = {mappings.get(position - start) for position in positions}
if len(names) == 1 and None not in names:
(initial if index == current_index else later)[index] = next(iter(names))
return initial, later
def maverick_pronoun_speakers(paragraphs, speaker_names, model=None,
device="cpu", target_pronouns=None,
return_stages=False):
"""Resolve reporting-clause pronouns to a supplied cast of entity names.
`model` accepts an object with Maverick's `predict(text)` API for tests.
A missing or ambiguous mapping is intentionally omitted.
"""
if target_pronouns is None:
target_pronouns = {
paragraph["index"]: speech_tag_pronouns(paragraph["text"])
for paragraph in paragraphs
}
if not speaker_names or not any(target_pronouns.values()):
return ({}, {}) if return_stages else {}
if model is None:
model = load_maverick(device)
initial, later = _maverick_resolutions(
paragraphs, speaker_names, model, target_pronouns,
)
return (initial, later) if return_stages else {**later, **initial}
def quotation_continues(previous_paragraph, paragraph):
"""Detect a new paragraph beginning inside the previous open quotation."""
return (previous_paragraph.count('"') % 2 == 1
and paragraph.startswith('"'))
def update_entity_relevance(scores, paragraph):
"""Add five once for a named reference, otherwise decay one to zero."""
for entity, score in scores.items():
mentioned = re.search(r"(?<!\w)" + re.escape(entity) + r"(?!\w)",
paragraph, re.IGNORECASE)
scores[entity] = score + 5 if mentioned else max(0, score - 1)
def highest_relevance_except(scores, previous_speaker):
"""Choose a unique top scorer other than the previous dialogue entity."""
candidates = {entity: score for entity, score in scores.items()
if entity != previous_speaker}
if not candidates:
return "Other"
highest = max(candidates.values())
winners = [entity for entity, score in candidates.items()
if score == highest]
return winners[0] if len(winners) == 1 else "Other"
def evaluate_attributions(paragraphs, expected_by_index, italic_word_indexes,
dialogue_rule=identify_dialogue, speaker_names=(),
maverick_model=None, device="cpu"):
"""Apply pronoun, continuation, and relevance rules, then score labels."""
dialogue = [
dialogue_rule(
paragraph["text"], paragraph["word_start"], italic_word_indexes,
paragraph["index"],
)
for paragraph in paragraphs
]
named_speakers = {
paragraph["index"]: speaker
for paragraph, is_dialogue in zip(paragraphs, dialogue)
if is_dialogue
if (speaker := named_speech_tag_speaker(
paragraph["text"], speaker_names
)) is not None
}
targets = {
paragraph["index"]: speech_tag_pronouns(paragraph["text"])
for paragraph, is_dialogue in zip(paragraphs, dialogue)
if is_dialogue and paragraph["index"] not in named_speakers
}
initial, later = maverick_pronoun_speakers(
paragraphs, speaker_names, model=maverick_model, device=device,
target_pronouns=targets, return_stages=True,
)
def predict_with(resolved):
relevance = {name: 0 for name in dict.fromkeys(speaker_names)
if name != "Other"}
predictions = []
previous_speaker = None
for position, (paragraph, is_dialogue) in enumerate(zip(paragraphs, dialogue)):
update_entity_relevance(relevance, paragraph["text"])
if not is_dialogue:
prediction = None
elif paragraph["index"] in named_speakers:
prediction = named_speakers[paragraph["index"]]
elif (position > 0 and dialogue[position - 1]
and quotation_continues(paragraphs[position - 1]["text"],
paragraph["text"])):
prediction = predictions[-1]
elif paragraph["index"] in resolved:
prediction = resolved[paragraph["index"]]
else:
prediction = highest_relevance_except(relevance, previous_speaker)
predictions.append(prediction)
if is_dialogue:
previous_speaker = prediction
return predictions
first_predictions = predict_with(initial)
upgrades = {index: name for index, name in later.items()
if first_predictions[index] == "Other"}
predictions = predict_with({**initial, **upgrades}) if upgrades else first_predictions
counts = {key: 0 for key in (
"correct_assignments", "incorrect_assignments", "missed_dialogue",
"extra_dialogue", "correct_non_dialogue",
)}
predicted_on_unlabeled = 0
for index, prediction in enumerate(predictions):
if index not in expected_by_index:
predicted_on_unlabeled += prediction is not None
continue
expected = expected_by_index[index]
if expected is None:
key = "correct_non_dialogue" if prediction is None else "extra_dialogue"
elif prediction is None:
key = "missed_dialogue"
elif prediction == expected:
key = "correct_assignments"
else:
key = "incorrect_assignments"
counts[key] += 1
expected_dialogue = sum(entity is not None for entity in expected_by_index.values())
return predictions, AttributionStats(
total_paragraphs=len(paragraphs),
labeled_paragraphs=len(expected_by_index),
unlabeled_paragraphs=len(paragraphs) - len(expected_by_index),
expected_dialogue=expected_dialogue,
expected_non_dialogue=len(expected_by_index) - expected_dialogue,
predicted_on_unlabeled=predicted_on_unlabeled,
**counts,
)

120
book-processing/epub_to_chapters.py
View File

@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""
EPUB -> a TOC-ordered folder of chapter TXT files.
EPUB -> a TOC-ordered book folder with chapter files in content/.
Usage:
python epub_to_chapters.py book.epub
@ -10,12 +10,18 @@ Usage:
Output:
output_directory/
TOC.txt
CONVERSION_REPORT.txt
content/
Dedication.txt
Dedication.rsvp
Chapter 1.txt
Chapter 1.rsvp
Chapter 2.txt
Chapter 2.rsvp
...
Each line of TOC.txt is both a section title and the basename of its text file.
Each line of TOC.txt is both a section title and the basename of its text file
inside content/.
The output files contain only section text. No chapter heading is added.
Strategy:
@ -36,6 +42,7 @@ from pathlib import Path
from urllib.parse import unquote, urlsplit
from bs4 import BeautifulSoup, NavigableString
from attribution import dialogue_word_indexes
try:
from ebooklib import epub, ITEM_DOCUMENT
@ -95,12 +102,17 @@ FAT_FORBIDDEN_FILENAME_CHARACTERS = re.compile(r'[<>:"/\\|?*\x00-\x1f]')
STYLE_ITALIC = 0x01
STYLE_BOLD = 0x02
# Extraction-only CSS marker. It is applied to text before sidecar indexes are
# built, and is never written as a reader style flag.
STYLE_SMALL_CAPS = 0x04
STYLE_MARKER = "\x01"
SIDECAR_MAGIC = b"RSVP"
# Version 2 activates timing modifiers. Version 1 sidecars remain readable by
# the firmware, but their reserved timing bytes are required to be zero.
SIDECAR_VERSION = 2
# Version 3 adds a semantic word-color field to each sparse record. The
# firmware also reads existing version 1 and 2 sidecars.
SIDECAR_VERSION = 3
SIDECAR_HEADER = struct.Struct("<4sBBQI")
WORD_COLOR_NONE = 0
WORD_COLOR_OTHER = 1
CSS_RULE = re.compile(r"([^{}]+)\{([^{}]*)\}")
CSS_CLASS = re.compile(r"\.([A-Za-z_-][A-Za-z0-9_-]*)")
WORD_LOOKUP_KEY = re.compile(r"^[^A-Za-z]*([A-Za-z]+(?:['-][A-Za-z]+)*)[^A-Za-z]*$")
@ -108,7 +120,6 @@ MAX_SECTION_BYTES = 1024 * 1024
MAX_SECTION_WORDS = 150_000
MAX_TOKEN_BYTES = 65_535
MAX_SECTION_NAME_BYTES = 200
RESERVED_SECTION_NAMES = {"toc", "conversion_report"}
TIMING_NONE = 0
TIMING_DOUBLE = 2
@ -272,6 +283,12 @@ def append_styled_text(pieces, node):
text = str(node)
if not text:
return
parent = node.parent
while parent is not None:
if parent.attrs.get("data-rsvp-small-caps") == "1":
text = text.upper()
break
parent = parent.parent
flags = style_flags_for_text_node(node)
pieces.extend((STYLE_MARKER, chr(ord("0") + flags), text,
STYLE_MARKER, "0"))
@ -520,38 +537,51 @@ def encode_varint(value):
return bytes(encoded)
def style_sidecar_bytes(text, style_records):
"""Return validated sparse style and timing metadata for a chapter."""
def style_sidecar_bytes(text, style_records, color_word_indexes=()):
"""Return sparse style, timing, and word-color metadata for a chapter."""
word_count = sum(1 for _ in re.finditer(r"\S+", text))
records = {}
previous_style_index = -1
for word_index, style_flags, timing_modifier in style_records:
if (word_index < 0 or word_index >= word_count or
style_flags & ~(STYLE_ITALIC | STYLE_BOLD) or
timing_modifier not in {TIMING_NONE, TIMING_DOUBLE, TIMING_TRIPLE} or
(style_flags == 0 and timing_modifier == TIMING_NONE) or
word_index <= previous_style_index):
raise ValueError("invalid style sidecar record")
records[word_index] = [style_flags, timing_modifier, WORD_COLOR_NONE]
previous_style_index = word_index
for word_index in color_word_indexes:
if word_index < 0 or word_index >= word_count:
raise ValueError("invalid word-color sidecar record")
records.setdefault(word_index, [0, TIMING_NONE, WORD_COLOR_NONE])[2] = (
WORD_COLOR_OTHER
)
payload = bytearray(SIDECAR_HEADER.pack(
SIDECAR_MAGIC,
SIDECAR_VERSION,
0,
source_fingerprint(text),
len(style_records),
len(records),
))
previous_index = 0
for record_number, (word_index, style_flags, timing_modifier) in enumerate(style_records):
if word_index < 0 or style_flags & ~(STYLE_ITALIC | STYLE_BOLD):
raise ValueError("invalid style sidecar record")
if timing_modifier not in {TIMING_NONE, TIMING_DOUBLE, TIMING_TRIPLE}:
raise ValueError("invalid style sidecar record")
if style_flags == 0 and timing_modifier == TIMING_NONE:
raise ValueError("invalid style sidecar record")
if record_number and word_index <= previous_index:
raise ValueError("style records must have ascending word indexes")
for record_number, (word_index, record) in enumerate(sorted(records.items())):
delta = word_index if record_number == 0 else word_index - previous_index
payload.extend(encode_varint(delta))
payload.extend((style_flags, timing_modifier))
payload.extend(record)
previous_index = word_index
return bytes(payload)
def write_style_sidecar(output_file, text, style_records):
"""Write validated sparse style and timing metadata beside a chapter."""
def write_style_sidecar(output_file, text, style_records, color_word_indexes=()):
"""Write validated sparse word metadata beside a chapter."""
output_file.write_bytes(style_sidecar_bytes(text, style_records))
output_file.write_bytes(style_sidecar_bytes(
text, style_records, color_word_indexes
))
def output_section_name(title, number, used_names):
@ -846,7 +876,7 @@ def find_document(href, by_path, by_filename):
# ---------------------------------------------------------------------------
def style_flags_from_css(declarations):
"""Return supported italic/bold flags from a CSS declaration block."""
"""Return supported EPUB text-style flags from a CSS declaration block."""
flags = 0
for property_name, value in re.findall(
@ -866,6 +896,10 @@ def style_flags_from_css(declarations):
flags |= STYLE_ITALIC
if "bold" in value or re.search(r"\b[6-9]00\b", value):
flags |= STYLE_BOLD
if property_name in {"font-variant", "font-variant-caps", "font"} and re.search(
r"\b(?:all-)?small-caps\b", value
):
flags |= STYLE_SMALL_CAPS
return flags
@ -903,6 +937,9 @@ def apply_css_styles(soup, class_styles):
flags = style_flags_from_css(tag.get("style", ""))
for class_name in tag.get("class", []):
flags |= class_styles.get(class_name, 0)
if flags & STYLE_SMALL_CAPS:
tag["data-rsvp-small-caps"] = "1"
flags &= ~STYLE_SMALL_CAPS
if flags:
tag["data-rsvp-style-flags"] = str(flags)
@ -1442,14 +1479,9 @@ def convert_epub(epub_path, output_dir, overwrite=False):
"Choose an empty directory or pass --overwrite."
)
sys.exit(1)
for output_file in output_dir.iterdir():
if output_file.is_file() and (
output_file.suffix in {".txt", ".rsvp"} or
output_file.name == "CONVERSION_REPORT.txt"
):
output_file.unlink()
output_dir.mkdir(parents=True, exist_ok=True)
content_dir = output_dir / "content"
print(f"Reading: {epub_path}")
@ -1561,6 +1593,11 @@ def convert_epub(epub_path, output_dir, overwrite=False):
print(f"Error: {exc}")
sys.exit(1)
for chapter in chapters:
chapter["color_word_indexes"] = dialogue_word_indexes(
chapter["text"], chapter["style_records"]
)
print(
f"Chapters to write: {len(chapters)}"
)
@ -1573,7 +1610,7 @@ def convert_epub(epub_path, output_dir, overwrite=False):
toc_entries = []
prepared_sections = []
used_names = set(RESERVED_SECTION_NAMES)
used_names = set()
for number, chapter in enumerate(
chapters,
@ -1597,7 +1634,8 @@ def convert_epub(epub_path, output_dir, overwrite=False):
print(f"Error: {title!r} exceeds reader limits: {exc}")
sys.exit(1)
sidecar_bytes = style_sidecar_bytes(
chapter["text"], chapter["style_records"]
chapter["text"], chapter["style_records"],
chapter["color_word_indexes"],
)
if len(text_bytes) > MAX_SECTION_BYTES or len(sidecar_bytes) > MAX_SECTION_BYTES:
print(
@ -1606,15 +1644,25 @@ def convert_epub(epub_path, output_dir, overwrite=False):
)
sys.exit(1)
if overwrite and content_dir.is_dir():
for output_file in content_dir.iterdir():
if output_file.is_file() and output_file.suffix in {".txt", ".rsvp"}:
output_file.unlink()
content_dir.mkdir(exist_ok=True)
written_sections = []
for number, (title, chapter) in enumerate(prepared_sections, start=1):
output_file = output_dir / f"{title}.txt"
sidecar_file = output_dir / f"{title}.rsvp"
output_file = content_dir / f"{title}.txt"
sidecar_file = content_dir / f"{title}.rsvp"
output_file.write_text(chapter["text"], encoding="utf-8")
write_style_sidecar(sidecar_file, chapter["text"], chapter["style_records"])
written_sections.append((output_file.name, chapter["text"]))
write_style_sidecar(
sidecar_file, chapter["text"], chapter["style_records"],
chapter["color_word_indexes"],
)
written_sections.append((f"content/{output_file.name}", chapter["text"]))
print(
f"{number:4}. {title} -> {output_file.name}, {sidecar_file.name}"
f"{number:4}. {title} -> content/{output_file.name}, "
f"content/{sidecar_file.name}"
)
toc_file = output_dir / "TOC.txt"

321
book-processing/label_dialogue.py Normal file
View File

@ -0,0 +1,321 @@
#!/usr/bin/env python3
"""Interactively label dialogue paragraphs in converted chapter text.
Usage:
python3 book-processing/label_dialogue.py Chapter.txt
python3 book-processing/label_dialogue.py Chapter1.txt Chapter2.txt \
--output dialogue-labels.json
The script uses quotation marks in the text and italic word flags in matching
RSVP sidecars to find candidate paragraphs. It records labels by chapter and
paragraph index, with entity names as the stable output identity.
"""
import argparse
import json
import re
import struct
import sys
from pathlib import Path
SIDECAR_HEADER = struct.Struct("<4sBBQI")
SIDECAR_MAGIC = b"RSVP"
SIDECAR_VERSIONS = {1, 2, 3}
STYLE_ITALIC = 0x01
STYLE_BOLD = 0x02
TIMING_MODIFIERS = {0, 2, 3}
LABEL_FORMAT = "dialogue-attribution-labels-v1"
# Single quotes are normalized to apostrophes by the converter. Require a
# pair that is not attached to word characters to avoid treating contractions
# such as don't as dialogue.
PAIRED_SINGLE_QUOTES = re.compile(r"(?<!\w)'[^'\n]+'(?!\w)")
WORD_TOKEN = re.compile(r"\b[\w’'-]+\b", re.UNICODE)
class SidecarError(ValueError):
"""A malformed, unsupported, or stale RSVP sidecar."""
def source_fingerprint(text):
"""Return the converter's stable 64-bit FNV-1a fingerprint."""
value = 0xCBF29CE484222325
for byte in text.encode("utf-8"):
value ^= byte
value = (value * 0x100000001B3) & 0xFFFFFFFFFFFFFFFF
return value
def read_varint(data, offset):
"""Read an unsigned LEB128 integer, returning its value and new offset."""
value = 0
shift = 0
while offset < len(data) and shift <= 28:
byte = data[offset]
offset += 1
if shift == 28 and byte > 0x0F:
raise SidecarError("sidecar word-index varint exceeds 32 bits")
value |= (byte & 0x7F) << shift
if not byte & 0x80:
return value, offset
shift += 7
raise SidecarError("invalid or oversized sidecar word-index varint")
def read_italic_word_indexes(sidecar_path, text):
"""Read italic word indexes after validating the sidecar against text."""
data = sidecar_path.read_bytes()
if len(data) < SIDECAR_HEADER.size:
raise SidecarError("sidecar is shorter than its header")
magic, version, reserved, fingerprint, record_count = SIDECAR_HEADER.unpack_from(data)
if magic != SIDECAR_MAGIC:
raise SidecarError("sidecar magic is not RSVP")
if version not in SIDECAR_VERSIONS:
raise SidecarError(f"unsupported sidecar version {version}")
if reserved != 0:
raise SidecarError("sidecar reserved header byte is not zero")
if fingerprint != source_fingerprint(text):
raise SidecarError("sidecar fingerprint does not match the chapter text")
word_count = sum(1 for _ in re.finditer(r"\S+", text))
if record_count > word_count:
raise SidecarError("sidecar has more records than chapter words")
italic_indexes = set()
offset = SIDECAR_HEADER.size
previous_index = 0
for record_number in range(record_count):
delta, offset = read_varint(data, offset)
record_bytes = 3 if version == 3 else 2
if offset + record_bytes > len(data):
raise SidecarError("sidecar ends inside a style record")
word_index = delta if record_number == 0 else previous_index + delta
flags, timing_modifier = data[offset], data[offset + 1]
color_flag = data[offset + 2] if version == 3 else 0
offset += record_bytes
if record_number and delta == 0:
raise SidecarError("sidecar word indexes are not strictly increasing")
if word_index >= word_count:
raise SidecarError("sidecar word index exceeds chapter word count")
if flags & ~(STYLE_ITALIC | STYLE_BOLD):
raise SidecarError("sidecar contains unknown style flags")
if timing_modifier not in TIMING_MODIFIERS:
raise SidecarError("sidecar contains an invalid timing modifier")
if version == 1 and timing_modifier != 0:
raise SidecarError("version 1 sidecar has a timing modifier")
if color_flag not in (0, 1):
raise SidecarError("sidecar contains an invalid word color flag")
if flags == 0 and timing_modifier == 0 and color_flag == 0:
raise SidecarError("sidecar contains an empty style record")
if flags & STYLE_ITALIC:
italic_indexes.add(word_index)
previous_index = word_index
if offset != len(data):
raise SidecarError("sidecar has trailing bytes")
return italic_indexes
def paragraphs_with_word_indexes(text):
"""Split blank-line-delimited paragraphs and report each word start index."""
paragraphs = []
word_start = 0
for raw_paragraph in re.split(r"\r?\n[ \t]*\r?\n+", text):
paragraph = raw_paragraph.strip()
if not paragraph:
continue
word_count = sum(1 for _ in re.finditer(r"\S+", paragraph))
paragraphs.append({
"index": len(paragraphs),
"text": paragraph,
"word_start": word_start,
"word_count": word_count,
})
word_start += word_count
return paragraphs
def paragraph_has_quotes(text):
"""Return whether a paragraph contains a likely dialogue quote marker."""
return '"' in text or bool(PAIRED_SINGLE_QUOTES.search(text))
def paragraph_has_all_caps_word(text):
"""Return whether a paragraph contains a word with 2+ uppercase letters."""
for token in WORD_TOKEN.findall(text):
letters = "".join(character for character in token if character.isalpha())
if len(letters) >= 2 and letters.isupper():
return True
return False
def is_candidate(paragraph, italic_indexes):
if paragraph_has_quotes(paragraph["text"]) or paragraph_has_all_caps_word(
paragraph["text"]
):
return True
start = paragraph["word_start"]
end = start + paragraph["word_count"]
return any(start <= index < end for index in italic_indexes)
def render_context(paragraphs, target_index, radius):
start = max(0, target_index - radius)
end = min(len(paragraphs), target_index + radius + 1)
for index in range(start, end):
label = "DIALOGUE" if index == target_index else "context"
print(f"[{label} paragraph {index + 1}]")
print(paragraphs[index]["text"])
print()
def save_labels(output_path, entities, chapters):
payload = {
"format": LABEL_FORMAT,
"entities": {str(number): name for number, name in sorted(entities.items())},
"chapters": chapters,
}
output_path.parent.mkdir(parents=True, exist_ok=True)
temporary_path = output_path.with_name(output_path.name + ".tmp")
temporary_path.write_text(
json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8",
)
temporary_path.replace(output_path)
def label_candidate(paragraphs, paragraph_index, entities):
radius = 1
while True:
render_context(paragraphs, paragraph_index, radius)
if entities:
print("Known entities: " + ", ".join(
f"{number}={name}" for number, name in sorted(entities.items())
))
print("Enter 0 for more context, 1-99 for a known entity, a name, or empty/null to skip.")
answer = input("Attribution: ").strip()
if answer == "0":
if radius >= len(paragraphs):
print("No additional chapter context is available.\n")
radius += 1
continue
if not answer or answer.casefold() == "null":
return None
if answer.isdecimal():
number = int(answer)
if 1 <= number <= 99 and number in entities:
return entities[number]
print("That number is not a previously assigned entity label.\n")
continue
known_number = next(
(number for number, name in entities.items()
if name.casefold() == answer.casefold()),
None,
)
if known_number is not None:
print(f"Using {known_number}={entities[known_number]}.\n")
return entities[known_number]
free_number = next((number for number in range(1, 100) if number not in entities), None)
if free_number is None:
print("All entity labels 1-99 are in use. Choose an existing label or skip.\n")
continue
entities[free_number] = answer
print(f"Assigned {free_number}={answer}.\n")
return answer
def load_chapter(path):
# Preserve newline bytes because the sidecar fingerprint covers the exact
# UTF-8 chapter text written by the converter.
text = path.read_bytes().decode("utf-8")
sidecar_path = path.with_suffix(".rsvp")
if sidecar_path.exists():
italic_indexes = read_italic_word_indexes(sidecar_path, text)
else:
print(f"Warning: {sidecar_path} is missing; italics will not be detected.", file=sys.stderr)
italic_indexes = set()
paragraphs = paragraphs_with_word_indexes(text)
if paragraphs and paragraphs[-1]["word_start"] + paragraphs[-1]["word_count"] != sum(
1 for _ in re.finditer(r"\S+", text)
):
raise ValueError(f"paragraph splitting did not align with word indexes in {path}")
return text, paragraphs, italic_indexes
def parse_args(argv=None):
parser = argparse.ArgumentParser(
description="Label quoted or italicized paragraphs with dialogue entities."
)
parser.add_argument("chapters", nargs="+", type=Path, help="chapter .txt files")
parser.add_argument(
"-o", "--output", type=Path, default=Path("dialogue-labels.json"),
help="output JSON file (default: dialogue-labels.json)",
)
return parser.parse_args(argv)
def main(argv=None):
args = parse_args(argv)
chapters = []
entities = {}
try:
for chapter_path in args.chapters:
if chapter_path.suffix.lower() != ".txt":
raise ValueError(f"expected a .txt chapter file: {chapter_path}")
if not chapter_path.is_file():
raise ValueError(f"chapter file does not exist: {chapter_path}")
text, paragraphs, italic_indexes = load_chapter(chapter_path)
chapter_result = {
"file": str(chapter_path.resolve()),
"paragraphs": [],
}
chapters.append(chapter_result)
save_labels(args.output, entities, chapters)
candidate_count = sum(
is_candidate(paragraph, italic_indexes) for paragraph in paragraphs
)
print(f"\n{chapter_path}: {candidate_count} candidate dialogue paragraphs")
for paragraph in paragraphs:
if not is_candidate(paragraph, italic_indexes):
continue
entity = label_candidate(paragraphs, paragraph["index"], entities)
chapter_result["paragraphs"].append({
"paragraph_index": paragraph["index"] + 1,
"text": paragraph["text"],
"entity": entity,
})
save_labels(args.output, entities, chapters)
save_labels(args.output, entities, chapters)
except (OSError, UnicodeError, SidecarError, ValueError) as exc:
print(f"Error: {exc}", file=sys.stderr)
return 2
except (EOFError, KeyboardInterrupt):
save_labels(args.output, entities, chapters)
print(f"\nSaved labels so far to {args.output}", file=sys.stderr)
return 130
print(f"\nSaved paragraph labels to {args.output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

5
book-processing/requirements.txt
View File

@ -2,3 +2,8 @@ ebooklib==0.20
beautifulsoup4==4.15.0
english-words==2.0.2
wordfreq==3.1.1
maverick-coref==1.0.7
# Transformers requires PyTorch 2.6+ when reading checkpoint files.
torch==2.6.0
# Maverick was released against the Transformers 4 API.
transformers>=4.34,<5

337
book-processing/tests/fixtures/attribution/chapter-1/Chapter 1.json vendored Normal file
View File

@ -0,0 +1,337 @@
{
"format": "dialogue-attribution-labels-v1",
"entities": {
"1": "Other",
"2": "Kaladin",
"3": "Syl",
"4": "Shallan",
"5": "Pattern"
},
"chapters": [
{
"file": "/home/j2/tds3p-rsvp-reader/Wind and Truth/Chapter 1.txt",
"paragraphs": [
{
"paragraph_index": 1,
"text": "I should have known I was being watched. All my life, the signs were there.",
"entity": "Other"
},
{
"paragraph_index": 2,
"text": "- From Knights of Wind and Truth, page 1",
"entity": null
},
{
"paragraph_index": 5,
"text": "He stood at his window on the first morning of the month. Sunlight streamed into the room around him, wind tickling his hair. He shouldn't have felt good. Yes, he'd helped protect Urithiru- but that victory had come at an agonizing cost. Beyond that, Dalinar had made a deal with the enemy: in just ten days, the champion of Honor and the champion of Odium would decide the fate of all Roshar.",
"entity": null
},
{
"paragraph_index": 7,
"text": "That should have made him an anxious, stewing pot of nerves. Instead he tipped his head back, sun warm on his skin, and acknowledged that while he didn't feel great, someday he would feel great again.",
"entity": null
},
{
"paragraph_index": 13,
"text": "\"Can't you make yourself any size?\" he asked, checking his jacket cuffs.",
"entity": "Kaladin"
},
{
"paragraph_index": 14,
"text": "\"Within reason.\"",
"entity": "Syl"
},
{
"paragraph_index": 15,
"text": "\"Whose reason?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 16,
"text": "\"No idea,\" she said. \"Tried to get as big as a mountain once. It involved lots of grunting and thinking like rocks. Really big rocks. Biggest I could manage was a very small mountain- small enough to fit in this room, with the tip brushing the ceiling.\"",
"entity": "Syl"
},
{
"paragraph_index": 17,
"text": "\"Then you could be tall enough to tower over me,\" he said. \"Why do you usually make yourself shorter?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 18,
"text": "\"It just feels right,\" she said.",
"entity": "Syl"
},
{
"paragraph_index": 19,
"text": "\"That's your explanation for basically everything.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 20,
"text": "\"Yup!\" She poked him. He could barely feel it. Even at this size, she was insubstantial in the Physical Realm. \"Uniform? I thought you weren't going to wear one anymore.\"",
"entity": "Syl"
},
{
"paragraph_index": 21,
"text": "He hesitated, then pulled the jacket down at the bottom to smooth the wrinkles across the sides. \"It just feels right,\" he admitted, meeting her eyes in the mirror.",
"entity": "Kaladin"
},
{
"paragraph_index": 23,
"text": "\"Someone is having a good day,\" she said, poking him again.",
"entity": "Syl"
},
{
"paragraph_index": 24,
"text": "\"Bizarrely,\" Kaladin said. \"Considering.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 25,
"text": "\"At least the war is almost over,\" she said. \"One more contest. Nine days.\"",
"entity": "Syl"
},
{
"paragraph_index": 26,
"text": "True. If Dalinar won, Odium had agreed to withdraw from Alethkar and Herdaz- though he could keep other lands he controlled, like Iri and Jah Keved. If Odium won, they were forced to cede Alethkar to the enemy. Plus there was a greater cost. If Dalinar lost, he had to join Odium, become Fused, and help conquer the cosmere. Kaladin wanted to think that the Radiants wouldn't follow as well, but he wasn't certain. So many people thirsted for war, even without the influence of an Unmade. Storms, he'd felt it too.",
"entity": null
},
{
"paragraph_index": 27,
"text": "\"Syl,\" he said, dropping his smile. \"I'm sure more people are going to die. Perhaps people I care about, but I can't be there to help them. Dalinar will have to choose someone else to be champion and- \"",
"entity": "Kaladin"
},
{
"paragraph_index": 28,
"text": "\"Kaladin Stormblessed,\" she said, rising higher into the air, arms folded. Though she wore a fashionable havah, she left her hair white-blue, flowing free, waving and shifting in the wind. The ... nonexistent wind. \"Don't you dare talk yourself into being miserable.\"",
"entity": "Syl"
},
{
"paragraph_index": 29,
"text": "\"Or what?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 30,
"text": "\"Or I,\" she thundered, \"shall make silly faces at you. As I alone can.\"",
"entity": "Syl"
},
{
"paragraph_index": 31,
"text": "\"They aren't silly,\" he said, shivering.",
"entity": "Kaladin"
},
{
"paragraph_index": 32,
"text": "\"They're hilarious.\"",
"entity": "Syl"
},
{
"paragraph_index": 33,
"text": "\"Last time you made a tentacle come out of your forehead.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 34,
"text": "\"Highbrow comedy.\"",
"entity": "Syl"
},
{
"paragraph_index": 35,
"text": "\"Then it slapped me.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 36,
"text": "\"Punch line. Obviously. All the humans in the world, and I picked the one without a taste for refined humor.\"",
"entity": "Syl"
},
{
"paragraph_index": 38,
"text": "\"It does feel warm,\" he said, \"to have finally figured a few things out. To let go of the weight and step out from the shadow. I know darkness will return, but I think ... I think I'll be able to remember better than before.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 39,
"text": "\"Remember what?\"",
"entity": "Syl"
},
{
"paragraph_index": 40,
"text": "He Lashed himself upward, floating until he was eye level with her. \"That days like this exist too.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 42,
"text": "\"I wish I could show Teft,\" Kaladin said. \"I feel his loss like a hole in my own flesh, Syl.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 43,
"text": "\"I know,\" she said softly.",
"entity": "Syl"
},
{
"paragraph_index": 49,
"text": "And he would remember.",
"entity": null
},
{
"paragraph_index": 51,
"text": "Kaladin ...",
"entity": "Other"
},
{
"paragraph_index": 53,
"text": "\"Syl,\" Kaladin said, \"did you say my name?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 54,
"text": "\"Huh?\" she said, opening her eyes.",
"entity": "Syl"
},
{
"paragraph_index": 55,
"text": "Kaladin ...",
"entity": "Other"
},
{
"paragraph_index": 57,
"text": "I need your help. I'm so sorry ... to ask more of you ...",
"entity": "Other"
},
{
"paragraph_index": 58,
"text": "\"Tell me you hear that,\" Kaladin said to Syl.",
"entity": "Kaladin"
},
{
"paragraph_index": 59,
"text": "\"I feel ...\" She cocked her head. \"I feel something. On the wind.\"",
"entity": "Syl"
},
{
"paragraph_index": 60,
"text": "\"It's speaking to me,\" he said, one hand to his head.",
"entity": "Kaladin"
},
{
"paragraph_index": 61,
"text": "A storm is coming, Kaladin, the wind whispered. The worst storm ... I'm sorry ...",
"entity": "Other"
},
{
"paragraph_index": 63,
"text": "\"What did you hear?\" Syl asked.",
"entity": "Syl"
},
{
"paragraph_index": 64,
"text": "\"A warning,\" he said, frowning. \"Syl, is the wind ... alive?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 65,
"text": "\"Everything is alive.\"",
"entity": "Syl"
},
{
"paragraph_index": 69,
"text": "Indeed, life was largely about perspective.",
"entity": null
},
{
"paragraph_index": 70,
"text": "Like this strange structure: a hollow, rectangular block hundreds of feet tall, dominating Shadesmar's landscape. People- spren- lived along the inside walls, walking up and down them, ignoring conventions of gravity. Looking down along one of the inside walls could be stomach-churning unless you changed your perspective. Unless you convinced yourself that walking up and down that wall was normal. Whether a person was strong or not wasn't usually subject to debate, yet if gravity could be a matter of opinion ...",
"entity": null
},
{
"paragraph_index": 73,
"text": "She'd bonded two. One during her childhood. One as an adult. She'd hurt the first, and had suppressed the memory.",
"entity": null
},
{
"paragraph_index": 75,
"text": "Nearby, Pattern's head pulsed to a vibrant rhythm- always moving, always forming some new geometric display. Comparing the two broke Shallan's heart. She had done this to Testament by rejecting the bond after using her Shardblade to kill her mother.",
"entity": null
},
{
"paragraph_index": 78,
"text": "\"Why?\" Shallan asked. \"Why don't you hate me?\"",
"entity": "Shallan"
},
{
"paragraph_index": 79,
"text": "Pattern rested his hand on Shallan's shoulder. \"We both knew the danger, the sacrifice, in bonding to humans again.\"",
"entity": "Pattern"
},
{
"paragraph_index": 80,
"text": "\"I hurt her.\"",
"entity": "Shallan"
},
{
"paragraph_index": 81,
"text": "\"Yet here you are,\" Pattern said. \"Able to stand tall. Able to control the Surges. Able to protect this world.\"",
"entity": "Pattern"
},
{
"paragraph_index": 82,
"text": "\"She should hate me,\" Shallan whispered. \"But there is no vitriol in the way she holds my hand. No judgment in the way she remains with us.\"",
"entity": "Shallan"
},
{
"paragraph_index": 83,
"text": "\"Because the sacrifice was worth something, Shallan,\" Pattern said, uncharacteristically reserved. \"It worked. In the end you recovered, did better. I am still here. And remarkably, I am not even a little bit dead! I do not think you will kill me at all, Shallan! I am happy about that.\"",
"entity": "Pattern"
},
{
"paragraph_index": 84,
"text": "\"Can I heal her?\" Shallan asked. \"Maybe if I ... if I bond her again?\"",
"entity": "Shallan"
},
{
"paragraph_index": 85,
"text": "\"I think, after talking to Kelek ...\" Pattern said. \"I think you are still bonded to her.\"",
"entity": "Pattern"
},
{
"paragraph_index": 86,
"text": "\"But ...\" Shallan glanced over her shoulder at him. \"I broke the bond. That did this.\"",
"entity": "Shallan"
},
{
"paragraph_index": 87,
"text": "\"Some breaks are messy,\" Pattern said. \"A slice with a sharpened knife is clean; a slice with a dull one is ragged. Your break, done by a child without full Intent, is ragged. In some ways that makes it worse, but it does mean that some Connection between you two persists.\"",
"entity": "Pattern"
},
{
"paragraph_index": 88,
"text": "\"So ...\"",
"entity": "Shallan"
},
{
"paragraph_index": 89,
"text": "\"So no,\" Pattern said. \"I do not think that merely saying Words once more would heal her.\" His head pattern spun a little more slowly, as if he were contemplating something profound. \"These numbers are ... perplexing, Shallan. Strangely irrational, in a sequence I do not understand. I mean ... I mean that we are walking on unfamiliar ground. A better metaphor for you. Yes. Unfamiliar ground. In the deep past, deadeyes did not exist.\"",
"entity": "Pattern"
},
{
"paragraph_index": 91,
"text": "The result had been the deadeyes. The explanation might lie with Kelek, the very person Shallan had been sent to Lasting Integrity to kill. She squeezed Testament's hand. \"I'm going to help you,\" Shallan whispered. \"Whatever it takes.\"",
"entity": "Shallan"
},
{
"paragraph_index": 93,
"text": "\"Thank you,\" Shallan said. \"For coming to me when I was young. Thank you for protecting me. I still do not remember it all, but thank you.\"",
"entity": "Shallan"
},
{
"paragraph_index": 95,
"text": "\"Rest now,\" Shallan said, wiping her eyes and standing. \"I'm going to figure this out.\"",
"entity": "Shallan"
}
]
}
]
}

BIN
book-processing/tests/fixtures/attribution/chapter-1/Chapter 1.rsvp vendored Normal file
View File

Binary file not shown.

191
book-processing/tests/fixtures/attribution/chapter-1/Chapter 1.txt vendored Normal file
View File

@ -0,0 +1,191 @@
I should have known I was being watched. All my life, the signs were there.
- From Knights of Wind and Truth, page 1
Kaladin felt good.
Not great. Not after spending weeks hiding in an occupied city. Not after driving himself to physical and emotional exhaustion. Not after what had happened to Teft.
He stood at his window on the first morning of the month. Sunlight streamed into the room around him, wind tickling his hair. He shouldn't have felt good. Yes, he'd helped protect Urithiru- but that victory had come at an agonizing cost. Beyond that, Dalinar had made a deal with the enemy: in just ten days, the champion of Honor and the champion of Odium would decide the fate of all Roshar.
The scope of that was terrifying, yet Kaladin had stepped down as leader of the Windrunners. He'd said the proper Words, but had realized Words alone weren't enough. While Stormlight healed his body instantly, his soul needed time. So, if battle came, his friends would fight without him. And when the champions met atop Urithiru in ten days- nine, since the first day was underway- Kaladin wouldn't participate.
That should have made him an anxious, stewing pot of nerves. Instead he tipped his head back, sun warm on his skin, and acknowledged that while he didn't feel great, someday he would feel great again.
For today, that was enough.
He turned and strode to his closet, where he picked through stacks of civilian clothing neatly laundered and delivered this morning. The city was a mere two days free from occupation and the fate of the world approached, but Urithiru's washwomen soldiered on. None of the clothes appealed to him, and shortly he glanced at another option: a uniform sent by the quartermaster to replace the one Kaladin had ruined during the fighting. Leyten kept a rack of them in Kaladin's size.
Kaladin had stuck the uniform to the wall with a Lashing last night, after Teft's funeral, as a test. Urithiru was awake, with its own Bondsmith, making things ... different. His Lashings normally lasted minutes at best- yet here this one was, ten hours later, still going strong.
Syl poked her head into his room- past the hanging cloth doorway- without any thought for privacy. Today she appeared at full human size and wore a havah rather than her usual girlish dress. She'd recently learned how to color her dress, in this case mostly darker shades of blue with some bright violet embroidery on her sleeves.
As Kaladin fastened the last buttons on the high collar of his uniform jacket, Syl bounced over to stand behind him. Then she floated a foot or so into the air to look over his shoulder and examine him in the mirror.
"Can't you make yourself any size?" he asked, checking his jacket cuffs.
"Within reason."
"Whose reason?"
"No idea," she said. "Tried to get as big as a mountain once. It involved lots of grunting and thinking like rocks. Really big rocks. Biggest I could manage was a very small mountain- small enough to fit in this room, with the tip brushing the ceiling."
"Then you could be tall enough to tower over me," he said. "Why do you usually make yourself shorter?"
"It just feels right," she said.
"That's your explanation for basically everything."
"Yup!" She poked him. He could barely feel it. Even at this size, she was insubstantial in the Physical Realm. "Uniform? I thought you weren't going to wear one anymore."
He hesitated, then pulled the jacket down at the bottom to smooth the wrinkles across the sides. "It just feels right," he admitted, meeting her eyes in the mirror.
She grinned. And storm him, he couldn't help grinning back.
"Someone is having a good day," she said, poking him again.
"Bizarrely," Kaladin said. "Considering."
"At least the war is almost over," she said. "One more contest. Nine days."
True. If Dalinar won, Odium had agreed to withdraw from Alethkar and Herdaz- though he could keep other lands he controlled, like Iri and Jah Keved. If Odium won, they were forced to cede Alethkar to the enemy. Plus there was a greater cost. If Dalinar lost, he had to join Odium, become Fused, and help conquer the cosmere. Kaladin wanted to think that the Radiants wouldn't follow as well, but he wasn't certain. So many people thirsted for war, even without the influence of an Unmade. Storms, he'd felt it too.
"Syl," he said, dropping his smile. "I'm sure more people are going to die. Perhaps people I care about, but I can't be there to help them. Dalinar will have to choose someone else to be champion and- "
"Kaladin Stormblessed," she said, rising higher into the air, arms folded. Though she wore a fashionable havah, she left her hair white-blue, flowing free, waving and shifting in the wind. The ... nonexistent wind. "Don't you dare talk yourself into being miserable."
"Or what?"
"Or I," she thundered, "shall make silly faces at you. As I alone can."
"They aren't silly," he said, shivering.
"They're hilarious."
"Last time you made a tentacle come out of your forehead."
"Highbrow comedy."
"Then it slapped me."
"Punch line. Obviously. All the humans in the world, and I picked the one without a taste for refined humor."
He met her eyes, and her smile was still storming infectious.
"It does feel warm," he said, "to have finally figured a few things out. To let go of the weight and step out from the shadow. I know darkness will return, but I think ... I think I'll be able to remember better than before."
"Remember what?"
He Lashed himself upward, floating until he was eye level with her. "That days like this exist too."
She nodded firmly.
"I wish I could show Teft," Kaladin said. "I feel his loss like a hole in my own flesh, Syl."
"I know," she said softly.
If she'd been a human friend, she might have offered a hug. Syl didn't seem to understand physicality like a human did, though where she'd been born- Shadesmar, the Cognitive Realm- she had a substantial body. He had the sense she hadn't spent much time on that side. This realm suited her.
Dropping to the ground, Kaladin walked back to the window, wanting to feel the sunlight. Outside he saw the heights of the mountains, capped by snow. Wind blew across him, bringing with it fresh scents of clean, crisp air and a flock of windspren. Including those that made up his armor, who soared in around him. They stayed close, in case they were needed.
Storms, he'd been through so much so quickly. He felt echoes of an anger that had almost entirely consumed him at Teft's death. Worse, the feeling of nothingness as he fell ...
Dark days.
But days like this existed too.
And he would remember.
His armor spren laughed and danced out the window, but the wind lingered, playing with his hair. Then it calmed, still blowing across him, but no longer playful, more ... contemplative. All through his life, the wind had been there. He knew it almost like he did his hometown or his family. Familiar ...
Kaladin ...
He jumped, then glanced at Syl, who was walking through the room in a half dance, half stride, her eyes closed- as if moving to an inaudible beat.
"Syl," Kaladin said, "did you say my name?"
"Huh?" she said, opening her eyes.
Kaladin ...
Storms. There it was again.
I need your help. I'm so sorry ... to ask more of you ...
"Tell me you hear that," Kaladin said to Syl.
"I feel ..." She cocked her head. "I feel something. On the wind."
"It's speaking to me," he said, one hand to his head.
A storm is coming, Kaladin, the wind whispered. The worst storm ... I'm sorry ...
It was gone.
"What did you hear?" Syl asked.
"A warning," he said, frowning. "Syl, is the wind ... alive?"
"Everything is alive."
He gazed outward, waiting for the voice to return. It didn't. Just that crisp breeze- though now it didn't seem calm.
Now it seemed to be waiting for something.
Shallan lingered atop Lasting Integrity, the great fortress of the honorspren, thinking about all the people she'd been. The way she changed, based on perspective.
Indeed, life was largely about perspective.
Like this strange structure: a hollow, rectangular block hundreds of feet tall, dominating Shadesmar's landscape. People- spren- lived along the inside walls, walking up and down them, ignoring conventions of gravity. Looking down along one of the inside walls could be stomach-churning unless you changed your perspective. Unless you convinced yourself that walking up and down that wall was normal. Whether a person was strong or not wasn't usually subject to debate, yet if gravity could be a matter of opinion ...
She turned away from the heart of Lasting Integrity and walked along the very top of the wall. Looking outward to survey Shadesmar: rolling ocean of beads in one direction, jagged obsidian highlands- lined with crystalline trees- in the other. On the wall with her, an even more daunting sight: two spren with heads made of geometric lines, each wearing a robe of some too-stiff glossy black material.
Two spren.
She'd bonded two. One during her childhood. One as an adult. She'd hurt the first, and had suppressed the memory.
Shallan knelt before Testament, her original spren. The Cryptic sat with her back to the stone railing. The lines and pattern that made up her head were crooked, like broken twigs. In the center the lines were scratched and rough, as if someone had taken a knife to them. More telling, her pattern was almost frozen.
Nearby, Pattern's head pulsed to a vibrant rhythm- always moving, always forming some new geometric display. Comparing the two broke Shallan's heart. She had done this to Testament by rejecting the bond after using her Shardblade to kill her mother.
Testament reached out with a long-fingered hand, and Shallan- pained- took it. It gripped hers lightly, but Shallan had the sense that was all the strength Testament had. She responded to being a deadeye differently from Maya, who stood nearby with Adolin and Kelek. Maya had always seemed strong of body, in spite of being a deadeye. Spren broke in different ways, it appeared. Just like people.
Testament squeezed Shallan's hand, bearing no expression but that torpid motion of lines.
"Why?" Shallan asked. "Why don't you hate me?"
Pattern rested his hand on Shallan's shoulder. "We both knew the danger, the sacrifice, in bonding to humans again."
"I hurt her."
"Yet here you are," Pattern said. "Able to stand tall. Able to control the Surges. Able to protect this world."
"She should hate me," Shallan whispered. "But there is no vitriol in the way she holds my hand. No judgment in the way she remains with us."
"Because the sacrifice was worth something, Shallan," Pattern said, uncharacteristically reserved. "It worked. In the end you recovered, did better. I am still here. And remarkably, I am not even a little bit dead! I do not think you will kill me at all, Shallan! I am happy about that."
"Can I heal her?" Shallan asked. "Maybe if I ... if I bond her again?"
"I think, after talking to Kelek ..." Pattern said. "I think you are still bonded to her."
"But ..." Shallan glanced over her shoulder at him. "I broke the bond. That did this."
"Some breaks are messy," Pattern said. "A slice with a sharpened knife is clean; a slice with a dull one is ragged. Your break, done by a child without full Intent, is ragged. In some ways that makes it worse, but it does mean that some Connection between you two persists."
"So ..."
"So no," Pattern said. "I do not think that merely saying Words once more would heal her." His head pattern spun a little more slowly, as if he were contemplating something profound. "These numbers are ... perplexing, Shallan. Strangely irrational, in a sequence I do not understand. I mean ... I mean that we are walking on unfamiliar ground. A better metaphor for you. Yes. Unfamiliar ground. In the deep past, deadeyes did not exist."
It was what they'd learned, in part, from the honorspren and from Maya. The deadeyes- all of them except Testament- had been bonded to ancient Radiants before the Recreance. Together they'd rejected their oaths, humans and spren alike. They'd thought it would cause a painful, but survivable split. Instead, something had gone terribly wrong.
The result had been the deadeyes. The explanation might lie with Kelek, the very person Shallan had been sent to Lasting Integrity to kill. She squeezed Testament's hand. "I'm going to help you," Shallan whispered. "Whatever it takes."
Testament didn't respond, but Shallan leaned in, wrapping her arms around the Cryptic. Pattern's robe always felt hard, yet Testament's bent like cloth.
"Thank you," Shallan said. "For coming to me when I was young. Thank you for protecting me. I still do not remember it all, but thank you."
The Cryptic slowly, but deliberately, put her arms around Shallan and squeezed back.
"Rest now," Shallan said, wiping her eyes and standing. "I'm going to figure this out."

1089
book-processing/tests/fixtures/attribution/prologue/Prologue- To Live.json vendored Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
book-processing/tests/fixtures/attribution/prologue/Prologue- To Live.rsvp vendored Normal file
View File

Binary file not shown.

618
book-processing/tests/fixtures/attribution/prologue/Prologue- To Live.txt vendored Normal file
View File

@ -0,0 +1,618 @@
SEVEN AND A HALF YEARS AGO
Gavilar Kholin was on the verge of immortality.
He merely had to find the right Words.
He walked a circle around the nine Honorblades, driven point-first into the stone ground. The air stank of burned flesh; he'd attended enough funeral pyres to know that scent intimately, though these bodies hadn't been burned after the fighting but during it.
"They call it Aharietiam," he said, trailing around the Blades, letting his hand linger on each one. When he became a Herald, would his Blade become like these, imbued with power and lore? "The end of the world. Was it a lie?"
Many who name it such believed what they said, the Stormfather replied.
"And the owners of these?" he said, gesturing to the Blades. "What did the Heralds believe?"
If they had been entirely truthful, the Stormfather said, then I would not be seeking a new champion.
Gavilar nodded. "I swear to serve Honor and Roshar as its Herald. Better than these did."
These words are not accepted, the Stormfather said. You will never find them at random, Gavilar.
He would try nonetheless. In becoming the most powerful man in the world, Gavilar had often accomplished what others thought impossible. He rounded the ring of Blades again, alone with them in the shadow of monolithic stones. After dozens of visits to this vision, he could name each and every Blade by its associated Herald. The Stormfather, however, continued to be reticent to share information.
No matter. He would have his prize. He ripped Jezrien's long, curved Blade from the stone and swung it, cutting the air. "Nohadon met and grew to know the Heralds."
Yes, the Stormfather admitted.
"They are in there, aren't they?" he said. "The correct Words are somewhere in The Way of Kings?"
Yes.
Gavilar had the entire book memorized- he'd taught himself to read years ago so he could search for secrets without revealing them to the women in his life. He tossed the Herald's Blade aside, letting it clang against the stone- which made the Stormfather hiss.
Gavilar mentally chided himself. This was just a vision, and these fake Blades were nothing to him, but he needed the Stormfather to think him pious and worthy at least for now. He took up Chana's Blade. He was fond of this one, as its ornamentation bifurcated the blade with a slit down the center. That long gap would be highly impractical for a normal sword. Here it was a symbol that this Blade was something incredible.
"Chanaranach was a soldier," he said, "and this is a soldier's Blade. Solid and straight, but with that little impossibility missing from the center." He held the Blade in front of him, examining its edge. "I feel I know them each so well. They are my colleagues, yet I could not pick them out of a crowd."
Your colleagues? Do not get ahead of yourself, Gavilar. Find the Words.
Those storming Words. The most important ones Gavilar would ever say. With them, he would become the Stormfather's champion- and, he had deduced, something more. Gavilar suspected he would be accepted into the Oathpact and ascend beyond mortality. He had not asked which Herald he would replace; it felt crass, and he did not want to appear crass before the Stormfather. He suspected, though, that he would replace Talenelat, the one who had not left his Blade.
Gavilar stabbed the sword back into the stone. "Let us return."
The vision ended immediately, and he was in the palace's second-floor study. Bookshelves, a quiet desk for reading, tapestries and carpets to dampen voices. Gavilar wore finery for the upcoming feast: regal robes more archaic than fashionable. Like his beard, the clothing stood out among the Alethi lighteyes. He wanted them to think of him as something ancient, beyond their petty games.
This room was technically Navani's, but it was his palace. People rarely looked for him here, and he needed a reprieve from little people with little problems. As he had time before his meetings, Gavilar selected a small book that listed the latest surveys of the region around the Shattered Plains. He was increasingly certain that place held an ancient unlocked Oathgate. Through it, Gavilar could find the mythical Urithiru, and there, ancient records.
He would find the right Words. He was close. So tantalizingly close to what all men secretly desired, but only ten had ever achieved. Eternal life, and a legacy that spanned millennia- because you could live to shape it.
It is not so grand as you think, the spren said. Which gave Gavilar pause. The Stormfather couldn't read his mind, could it? No. No, he'd tested that. It didn't know his deepest thoughts, his deepest plans. For if it did know his heart, it wouldn't be working with him.
"What isn't?" Gavilar asked, slipping the book back.
Immortality, the Stormfather said. It wears on men and women, weathering souls and minds. The Heralds are insane- afflicted with unnatural ailments unique to their ancient natures.
"How long did it take?" Gavilar asked. "For the symptoms to appear?"
Difficult to say. A thousand years, perhaps two.
"Then I will have that long to find a solution," Gavilar said. "A much more reasonable timeline than the mere century- with luck- afforded a mortal. Wouldn't you say?"
I have not promised you this boon. You guess it is what I offer, but I seek only a champion. Still, tell me, would you accept the cost of becoming a Herald? Everyone you know would be dust by the time you returned.
And here, the lie. "A king's duty is to his people," he said. "By becoming a Herald, I can safeguard Alethkar in a way that no previous monarch ever has. I can endure personal pain to accomplish this. 'If I should die,'" Gavilar added, quoting The Way of Kings, "'then I would do so having lived my life right. It is not the destination that matters, but how one arrives there.'"
These words are not accepted, the spren said. Guessing will not bring you to the Words, Gavilar.
Yes, well, the Words were in that volume somewhere. Sheltered among the self-righteous moralizing like a whitespine in the brambles. Gavilar Kholin was not a man accustomed to losing. People got what they expected. And he expected not just victory, but divinity.
The guard knocked softly. Was it time already? Gavilar called for Tearim to come in, and he did. The guard was wearing Gavilar's own Plate tonight.
"Sire," Tearim said, "your brother is here."
"What? Not Restares? How did Dalinar find me?"
"Spotted us standing watch, I suspect, Your Majesty."
Bother. "Let him in."
The guard withdrew. A second later Dalinar burst in, graceful as a three-legged chull. He slammed the door and bellowed, "Gavilar! I want to go talk to the Parshendi."
Gavilar took a long, deep breath. "Brother, this is a very delicate situation, and we don't want to offend them."
"I won't offend them," Dalinar grumbled. He wore his takama, the robe of the old-fashioned warrior's garb open to show his powerful chest- with some grey hairs. He pushed past Gavilar and threw himself into the seat by the desk.
That poor chair.
"Why do you even care about them, Dalinar?" Gavilar said, right hand to his forehead.
"Why do you?" Dalinar demanded. "This treaty, this sudden interest in their lands. What are you planning? Tell me."
Dear, blunt Dalinar. As subtle as a jug of Horneater white. And equally smart.
"Tell me straight," Dalinar continued. "Are you planning to conquer them?"
"Why would I be signing a treaty if that were my intent?"
"I don't know," Dalinar said. "I just ... I don't want to see anything happen to them. I like them."
"They're parshmen."
"I like parshmen."
"You've never noticed a parshman unless he was too slow to bring your drink."
"There's something about these ones," Dalinar said. "I feel a ... a kinship."
"That's foolish." Gavilar walked to the desk and leaned down beside his brother. "Dalinar, what's happening to you? Where is the Blackthorn?"
"Maybe he's tired," Dalinar said. "Or blinded. By the soot and ashes of the dead, constantly in his face ..."
Again Dalinar whined about the Rift? What an enormous hassle. Restares would be here soon, and then ... there was Thaidakar. So many knives to keep balanced perfectly on their tips, lest they slide and cut Gavilar. He couldn't deal with Dalinar having a crisis of conscience right now.
"Brother," Gavilar said, "what would Evi say if she saw you like this?"
It was a carefully sharpened spear, slipped expertly into Dalinar's gut. The man's fingers gripped the table, and he recoiled at her name.
"She would want you to stand as a warrior," Gavilar said softly. "And protect Alethkar."
"I ..." Dalinar whispered. "She ..."
Gavilar offered a hand and heaved his brother to his feet, then led him to the door. "Stand up straight."
Dalinar nodded, hand on the doorknob.
"Oh," Gavilar said. "And Brother? Follow the Codes tonight. There is something strange upon the winds."
The Codes said not to drink when battle might be imminent. Just a nudge to remind Dalinar that it was a feast, and that there was plenty of wine on hand. Though Dalinar still thought no one knew he'd killed Evi, Gavilar had found the truth, which let him use these subtle manipulations.
Dalinar was out the door a moment later, his lumbering, pliable brain likely focused on two things. First, what he'd done to Evi. Second, how to find something strong enough to make him forget about the first.
When Dalinar was off down the hallway, Gavilar waved Tearim close. The guard was one of the Sons of Honor, a group that was yet another knife Gavilar kept balanced, for they could never know he had outgrown their plans.
"Follow my brother," Gavilar said. "Subtly ensure that he gets something to drink; maybe lead him to my wife's secret stores."
"You had me do that a few months ago, sire," Tearim whispered back. "There's not much left, I'm afraid. He likes to share with his soldiers."
"Well, find him something," Gavilar replied. "I can let Restares and the others in when they arrive. Go."
The soldier bowed and followed Dalinar, Shardplate thumping. Gavilar shut the door firmly. When the Stormfather's voice pushed into his mind, he was not surprised.
He has potential you do not see, that one.
"Dalinar? Of course he does. If I can keep him pointed the right direction, he will burn down entire nations." Gavilar simply had to ply him with alcohol the rest of the time, so that he didn't burn down this nation.
He could be more than you think.
"Dalinar is a big, dumb, blunt instrument you apply to problems until they break," Gavilar said, then shivered, remembering seeing his brother approach across a battlefield. Soaked in blood. Eyes appearing to glow red within his helm, hungry for the life Gavilar lived ...
That ghost haunted him. Fortunately, both Dalinar's pain and his addiction made him easy enough to control.
Gavilar was soon interrupted by another knock. He answered the door and found nothing outside, until the Stormfather hissed a warning in his mind and he felt a sudden chill.
When he turned around, old Thaidakar was there. The Lord of Scars himself, a figure in an enveloping hooded cloak, tattered at the bottom. Storms.
"I was made promises," Thaidakar said, hood shadowing his face. "I've given you information, Gavilar, of the most valuable nature. In payment I requested a single man. When will you deliver Restares to me?"
"Soon," Gavilar said. "I am gaining his confidence first."
"It seems to me," Thaidakar said, "that you're less interested in our bargain, and more interested in your own motives. It seems to me that I directed you toward something valuable you've decided to keep. It seems to me that you play games."
"It seems to me," Gavilar said, stepping closer to the cloaked figure, "that you're not in a position to make demands. You need me. So why don't we just ... keep playing."
Thaidakar remained still for a moment. Then, with a sigh, he reached up with gloved hands and took down his hood. Gavilar froze- for despite their several interactions, he'd never before seen the man's face.
Thaidakar was made entirely of softly glowing white-blue light. He was younger than Gavilar had imagined- in his middle years, not the wizened elder he'd seemed. He had a large spike, also blue, through one eye. The point jutted out the back of his skull. Was he some kind of spren?
"Gavilar," Thaidakar said, "take care. You're not immortal yet, but you've begun to play with forces that rip mortals apart by their very axi."
"Do you know what they are?" Gavilar demanded, hungry. "The most important Words I'll ever speak?"
"No," Thaidakar said. "But listen: none of this is what you think it is. Deliver Restares to my agents, and I will help you recover the ancient powers."
"I've grown beyond that," Gavilar said.
"You can't 'grow beyond' the tide, Gavilar," Thaidakar replied. "You swim with it or get swept away. Our plans are already in motion. Though to be honest, I don't know that we did much. That tide was coming regardless."
Gavilar grunted. "Well, I intend to- "
He was cut off as Thaidakar transformed. His face melted into a simple floating sphere with some kind of arcane rune at the center. The cloak, body, and gloves vanished into wisps of smoke that evaporated away.
Gavilar stared. That ... that looked a lot like what he'd read of the powers of Lightweavers. Knights Radiant. Was Thaidakar- ?
"I know you're meeting Restares today," the sphere said, vibrating- it had no mouth. "Prepare him, then deliver him to my agents for questioning. Or else. That is my ultimatum, Gavilar. You would not like to be my enemy."
The sphere of light shrank and turned nearly transparent as it moved to the door, then bobbed down and vanished through the crack underneath.
"What was that?" Gavilar demanded of the Stormfather, unnerved.
Something dangerous, the spren replied in his mind.
"Radiant?"
No. Similar, but no.
Gavilar found himself trembling. Which was stupid. He was a storming king, soon to be a demigod. He had a destiny; he would not be unsettled by cheap tricks and vague threats. Still, he rested his hand against the desk and breathed deeply, his fingers disturbing scattered notes and diagrams from his wife's latest mechanical obsession. Not for the first time he wondered if Navani could crack this problem. He missed the way they'd once schemed. How long had it been since they'd all laughed together? He, Ialai, Navani, and Torol?
Unfortunately, this wasn't the kind of secret you shared. Ialai or Sadeas would seize the prize from him if they could- and Gavilar wouldn't blame them. Navani though ... would she try to take immortality for herself? Would she even see its value? She was so clever, so crafty in some ways. Yet when he spoke of his goals for a greater legacy, she got lost in the details. Refusing to think of the mountain because she worried about the placement of the foothills.
He regretted the distance between them. That coldness growing over- well, grown over- their relationship. Thinking of her sent a stab of pain into his heart. He should ...
Everyone you know will be dust by the time you return ...
Perhaps this way was best.
He had plans to mitigate the length of his absence from this world, but they might take several tries to perfect. So ... fewer attachments seemed better. To allow for a cleaner cut. Like one made with a Shardblade.
He bent his mind to his plans, and was well prepared by the time Restares arrived. The balding man didn't knock. He peeked in, nervously checking each corner before he slipped through the door. He was followed by a shadow: a tall, imperious Makabaki man with a birthmark on one cheek. Gavilar had instructed the servants to treat them as ambassadors, but he hadn't yet had a chance to speak with this second man, whom he didn't know.
The newcomer walked with a certain ... firmness. Rigidity. He wasn't a man who gave way. Not to wind, not to storm, and most certainly not to other people.
"Gavilar Kholin," the man said, offering neither a hand nor a bow. They locked stares. Impressive. Gavilar had expected ... well, someone more like Restares.
"Have a drink," Gavilar said, gesturing toward the bar.
"No," the man said. Without a thank-you or compliment. Interesting. Intriguing.
Restares scuttled over like a child offered sweets. Even now- after joining this newest incarnation of the Sons of Honor- Gavilar found Restares ... odd. The short, balding man sniffed at each of the wines. He had never trusted a drink in Gavilar's presence, but always tested them anyway. As if he wanted to find poison, to prove his paranoia was justified.
"Sorry," Restares said, wringing his hands as he hovered over the drinks. "Sorry. Not ... not thirsty today, Gavilar. Sorry."
Gavilar was close to tossing him aside and seizing control of the Sons of Honor. Except some of the others, like Amaram, respected him. Plus ... why was Thaidakar so interested in Restares? Surely he couldn't actually be someone important. Perhaps his tall friend was the true power. Could Gavilar have been kept in the dark for two years about something that vital?
"I'm glad you were willing to meet," Restares said. "Yes, um. Because, um. So ... Announcement. I have an announcement."
Gavilar frowned. "What is this?"
"I hear," Restares said, "that you're looking to, um, restore the Voidbringers?"
"You founded the Sons of Honor, Restares," Gavilar said, "to recover the ancient oaths and restore the Knights Radiant. Well, they vanished when the Voidbringers did. So if we bring back the Voidbringers, the powers should return."
More importantly, he thought, the Heralds will return from the land of the dead to lead us again.
Letting me usurp the position of one of them.
"No, no, no," Restares said, uncharacteristically firm. "I wanted the honor of men to return! I wanted us to explore what made those Radiants so grand. Before things went wrong." He ran his hand through his thinning hair. "Before ... I made them ... go wrong ..."
Restares wouldn't meet Gavilar's eyes. "We ... we should stop trying to restore the powers," Restares said, his voice wilting, and glanced to his stern friend- as if for support. "We can't ... afford another Return ..."
"Restares," Gavilar said, advancing on the little man. "What is wrong with you? You're talking about betraying everything we believe?" Or at least pretend to believe. Gavilar subtly placed himself so he loomed over Restares. "Have you heard of a man named Thaidakar?"
Restares looked up, his eyes widening.
"He wants to find you," Gavilar said. "I have protected you thus far. What is it he wants from you, Restares?"
"Secrets," Restares whispered. "The man ... can't abide ... anyone having secrets."
"What secrets?" Gavilar said firmly, making Restares cringe. "I've put up with your lies long enough. What is going on? What does Thaidakar want?"
"I know where she is hidden," Restares whispered. "Where her soul is. Ba-Ado-Mishram. Granter of Forms. The one who could rival Him. The one ... we betrayed."
Ba-Ado-Mishram? Why would Thaidakar care about an Unmade? It seemed such an oddly shaped piece of the puzzle. Gavilar opened his mouth to speak, but a hand squeezed his shoulder, fingers like a vise. Gavilar turned to see Restares's Makabaki friend standing behind him.
"What have you done?" the man asked, his voice icy. "Gavilar Kholin. What actions have you taken to achieve this goal of yours, the one that my friend mistakenly set you upon?"
"You have no idea," Gavilar said, meeting the stranger's eyes until the man finally released his grip.
Gavilar took a pouch from his pocket, then casually spilled a selection of spheres and gemstones onto the table. "I'm close. Restares, you must not lose your nerve now!"
The stranger stared, his lips parting. He reached toward one of the spheres that glowed with a dark, almost inverted violet light. Impossible light; a color that should not exist. As soon as the stranger's fingers drew close, he yanked them away, then turned wide eyes on Gavilar.
"You are a fool," the man said. "A terrible fool charging toward the highstorm with a stick, thinking to fight it. What have you done? Where did you get Voidlight?"
Gavilar smiled. None of them knew of the secret scholar he kept in reserve. A master of all things scientific. A man who was neither Ghostblood nor Son of Honor.
A man from another world.
"It is set in motion," Gavilar said, glancing at Restares. "And the project was a success."
Restares perked up. "It ... it was? Is that Light ..." He turned to his friend. "This could work, Nale! We could bring them back, then destroy them. It could work."
Nale. Oh, storms. Gavilar knew- but tried to ignore- that Restares pretended to be a Herald to impress the others. The little man didn't know Gavilar had become familiar with the Stormfather, who had told him the truth: that the Heralds had all long since died and gone to Braize.
So was this stranger pretending to be Nalan, Herald of Justice? He ... had the right look. Many of the depictions were of an imperious Makabaki man. And that birthmark ... it was strikingly similar to one on several of the older paintings.
But no. That was ridiculous. To believe that, one would have to believe that Restares- of all people- was a Herald.
The stranger tried to stare Gavilar down. Motionless, his expression cold. A monolith instead of a man. "This is far too dangerous."
Gavilar continued to hold his gaze. The world would bend to his desires. It always had before.
"But you are," the man eventually said, stepping back, "the king. Your will ... is law ... in this land."
"Yes," Gavilar said. "That is correct. Restares, I have more good news. We can move Voidlight from the storm to the Physical Realm. We can even carry it between here and Damnation, as you wanted."
"That's a way," Restares said, looking to Nale. "A way ... maybe to escape ..."
Nale waved to the spheres. "Being able to bring them back and forth from Braize doesn't mean anything. It's too close to be a relevant distance."
"It was impossible only a few short years ago," Gavilar said. "This is proof. The Connection is not severed, and the box allows for travel. Not yet as far as you'd like, but we must start the journey somewhere."
He wasn't certain why Restares was so eager to move Light around in Shadesmar. Thaidakar wanted this information as well. A way to transport Stormlight, and this new Voidlight, long distances. As he was contemplating that, Gavilar saw something. The door was cracked. An eye was peeking through.
Damnation. It was Navani. How much had she heard?
"Husband," she said, immediately pushing into the room. "There are guests missing you at the gathering. You seem to have lost track of time."
He smothered his anger at her spying, turning to Restares and his friend. "Gentlemen, I will need to excuse myself."
Restares again ran his hand through his wispy hair. "I want to know more of the project, Gavilar. Plus, you need to know that another of us is here tonight. I spotted her handiwork earlier."
Another one? Another Son of Honor.
No, he meant another Herald. Restares was growing more delusional.
"I have a meeting shortly with Meridas and the others," Gavilar said, calmly soothing Restares. "They should have more information for me. We can speak again after that."
"No," the Makabaki man growled. "I doubt we shall."
"There's more here, Nale!" Restares said, though he followed as Gavilar ushered the two of them from the room. "This is important! I want out. This is the only way ..."
Gavilar shut the door. Then turned to his wife. Damnation, she should know better than to interrupt him. She ...
Storms. The dress was beautiful, her face more so, even when angry. Staring at him with brilliant eyes, a fiery halo almost seeming to spread around her.
Once more, he considered.
Once more he rejected the idea.
If he was going to be a god, best to sever attachments. The sun could love the stars. But never as an equal.
Some time later, after he'd seen to Navani, Gavilar slipped away again. To his chambers this time, where he could confront what he'd learned.
"Tell me," he said, walking across the springy carpet to regard the tabletop map of Roshar. "Why is Thaidakar so interested in Ba-Ado-Mishram?"
The Stormfather formed a rippling beside Gavilar, vaguely in the shape of a person, but indistinct. Like the wavering in the air made by great heat on the stones.
She created your parshmen by accident, he said. Long ago, just before the Recreance, Mishram tried to rise up and replace Odium, giving the Voidbringers powers.
"Curious," Gavilar said. "And then?"
And then ... she fell. She was too small a being to uphold an entire people. It all came crashing down, and so some brave Radiants trapped Mishram in a gemstone to prevent her from destroying all of Roshar. A side effect created the parshmen.
Simple parshmen. They were Voidbringers. A delicious secret he'd pried out of the Stormfather some weeks ago. Gavilar strolled to the bookcase, where one of the new heating fabrials had been delivered to him by the scholar Rushur Kris. He took it from its cloth casing, weighing it.
He had found a way to ferry Voidspren through Shadesmar to this world using gemstones and aluminum boxes. Who would have thought Navani's pet area of study would be so useful? And if that conniving Axindweth eluded his grasp, he'd have to do the next part without her. He had his scholar, though in truth Gavilar was baffled by the Light he was creating ... Light that could somehow kill the Voidbringers? How had Vasher made-
He thought he heard a faint crackling sound from the Stormfather. Lightning? How cute.
"You've never challenged what I'm doing," Gavilar said. "I would have thought that returning the Voidbringers would be opposed to your very nature."
Opposition is sometimes needed, the Stormfather said. You will need someone to fight, should you become champion.
"Give it to me," Gavilar said. "Now. Make me a Herald. I need it."
The Stormfather turned a shimmering head in his direction. That was almost them.
"What, those?" Gavilar said. "A demand?"
So close. And so far.
Gavilar smiled, hefting the fabrial and thinking of the flamespren trapped inside. The Stormfather seemed increasingly suspicious, hostile. If things did go poorly ... could he trap the Stormfather himself in one of these?
Soon Amaram arrived with a small collection of people: two men, two women. One man was Amaram's lieutenant. The other three would be new important Sons of Honor recruits, invited to the feast and given exclusive time with the king after. It was an annoyance, but a worthy one. Gavilar identified the two women from the notes, but not the older man in robes. Who was he? A stormwarden? Amaram liked to keep them around, to teach him their script, which preserved some semblance of Vorin devotion. That was important to him.
Gavilar met each guest in turn, and as he reached the older man, something clicked. This was Taravangian, the king of Kharbranth. Famously a man of little consequence or aptitude. Gavilar glanced at Amaram. Surely they weren't going to invite him into their confidence- they should find the power who secretly ruled Kharbranth. Likely to be one of two women, per Gavilar's spy reports.
Amaram nodded. So, Gavilar gave his speech about ancient oaths and Radiants- of glories past and futures bright. It was a good speech, but beginning to grate on him. Once his words had inspired troops; now he spent his entire life in meetings. After finishing, he let everyone get something to drink.
"Meridas," Gavilar whispered, pulling Amaram aside. "These meetings are growing onerous. My experiment was a success. I have the weapon."
Amaram started, then spoke softly. "You mean ..."
"Yes, once we bring back the Voidbringers, we will have a new way to fight them."
"Or a new way to control them," Amaram whispered.
Well, that was new. Gavilar considered his friend, and the ambition those words implied. Good for you, Amaram.
"We must restore the Desolations," Gavilar said. "Whatever the cost. It's the only way."
"I agree," Amaram said. "Now more than ever." He hesitated. "My efforts with your daughter did not go well earlier. I thought we had an understanding."
"You simply need more time, my friend. To win her over."
Amaram hungered for the throne like Gavilar hungered for immortality. And perhaps Gavilar would reward Amaram with it. Elhokar certainly did not deserve to be king. He was exactly the opposite of the legacy Gavilar wanted.
He sent Amaram to talk to the others. Once they had enjoyed their drinks, Gavilar would give another short speech. Then he could be on to ... He frowned, noticing that one of the new recruits wasn't conversing with the others. The elderly man, Taravangian, was staring at the map of Roshar. The others laughed at something Amaram said. Taravangian didn't even look toward the sound.
Gavilar strode over, but before he could speak, Taravangian whispered, "Do you ever wonder about the lives we're giving them? The people we rule?"
Gavilar was unaccustomed to people- let alone strangers- addressing him with such familiarity. But then, this Taravangian saw himself as a king, and perhaps as Gavilar's equal. Laughable, when Taravangian ruled only one small city.
"I worry less about their lives now," Gavilar said, "and more about what is to come."
Taravangian nodded, appearing thoughtful. "That was an inspiring speech. Do you actually believe it?"
"Would I say it if I didn't?"
"Of course you would; a king will say whatever needs to be said. Wouldn't it be grand if that were always what he believed?" He looked to Gavilar, smiling. "Do you truly believe the Radiants can return?"
"Yes," Gavilar said. "I do."
"And you are not a fool," Taravangian said, musing. "So you must have good reasons."
Gavilar found himself revising his earlier opinion. A little king was still a king. Perhaps, among all of the dignitaries in the city tonight, here was one who might ... in the smallest way ... understand the demands placed on the man pressed between crown and throne.
"A danger is coming," Gavilar said softly, shocked at his own sincerity. "To this land. This world. An ancient danger."
Taravangian narrowed his eyes.
"It's not just a Desolation we must fear," Gavilar said. "They come. The Everstorm. The Night of Sorrows."
Taravangian, remarkably, grew pale.
He believed. Gavilar usually felt foolish when he tried to explain the true dangers that the Stormfather had shown him- the contest of champions for the fate of Roshar. He worried people would think him mad. Yet this man ... believed him?
"Where," Taravangian asked, "did you hear those words?"
"I don't know that you'd believe me if I told you."
"Will you believe me?" Taravangian asked. "Ten years ago, my mother died of her tumors. Frail, lying on her bed, with too many perfumes struggling to smother the stench of death. She gazed at me in her last moments ..." He met Gavilar's eyes. "And she whispered: 'I stand before him, above the world itself, and he speaks the truth. The Desolation is near ... The Everstorm. The Night of Sorrows.' Then she was gone."
"I've ... heard of this," Gavilar admitted. "The prophetic words of the dying ..."
"Where did you hear those words?" Taravangian asked, practically begging. "Please."
"I see visions," Gavilar said, frank. "Given me by the Almighty. So that we may prepare." He looked toward the map. "Heralds send that I may become the person I need to be to stop what is coming ..."
Let the Stormfather see sincerity in Gavilar. Storms ... suddenly Gavilar felt it. Standing there with this little king, he felt it. Never before- in all of this- had Gavilar ever suspected he might be inadequate to the task.
Perhaps, he thought, I should encourage Dalinar to resume his training. Remind him that he is a soldier. Gavilar had the distinct impression that before too long, he would need the Blackthorn again.
Someone is approaching your door outside, the Stormfather warned. One of the listeners. Eshonai. There is something about this one ...
One of the Parshendi? Gavilar shook himself. He dismissed Taravangian, Amaram, and the others- happy to be rid of that strange old man and his questioning eyes. The fellow was supposed to be unremarkable. Why did he unnerve Gavilar?
Eshonai entered as Amaram passed along his invitation. The conversation with the parshwoman went smoothly, with him manipulating her- and therefore her people. To prepare them for the role they would play.
After Gavilar grew weary at the feast once the treaty was signed, he retired to his rooms. He sank into a deep plush chair by his balcony, releasing a long sigh. Early in his career as a warlord, he'd never allowed himself the luxury of softness. He had mistakenly assumed that liking something soft would make him soft.
A common failing among men who wished to appear strong. It was not weakness to relax. By being so afraid of it, they gave simple things power over them.
The air shimmered in front of him.
"A full day," Gavilar said.
Yes.
"The first of many such," Gavilar continued. "I will be mounting an expedition back to the Shattered Plains soon. We can leverage my new treaty to obtain guides, have them lead us inward to the center. Toward Urithiru."
The Stormfather did not reply. Gavilar wasn't certain the spren could be said to have human mannerisms. Today though ... that turned-away posture, hinted at in the warping of the air ... that silence ...
"Do you regret choosing me?" Gavilar asked.
I regret the way I have treated you, the Stormfather said. I should not have been so accommodating. It has made you lazy.
"This is lazy?" Gavilar said, forcing amusement into his voice to hide his annoyance.
You do not reverence the position you seek, the Stormfather said. I feel ... you are not the champion I need. Maybe ... I've been wrong all this time.
"You said that you were charged with this task of finding a champion," Gavilar said. "By Honor."
That is true. I do not speak in human ways. But still, if you become a Herald, you will be tortured between Returns. Why is it this doesn't bother you?
Gavilar shrugged. "I will just give in."
What?
"Give in," Gavilar said, heaving himself out of his seat. "Why stay to be tortured and potentially lose my mind? I will give up each time and return immediately."
The Heralds stay in Damnation to seal the Voidbringers away. To prevent them from overrunning the world. They-
"The Heralds are the ten fools for that," Gavilar explained, pouring a drink from the carafe near his balcony. "If I cannot die, I will be the greatest king this world has ever known. Why lock away my knowledge and leadership?"
To stop the war.
"Why would I care to stop a war?" Gavilar asked, genuinely amused. "War is the path to glory, to training our soldiers to recover the Tranquiline Halls. My troops should be experienced, don't you think?" He turned back toward the shimmer, taking a sip of orange wine. "I don't fear these Voidbringers. Let them stay and fight. If they are reborn, then we will never run out of enemies to kill."
The Stormfather did not respond. And again Gavilar tried to read into the thing's posture. Was the Stormfather proud of him? Gavilar considered this an elegant solution; he was puzzled why the Heralds had never thought of it. Perhaps they were cowards.
Ah, Gavilar, the Stormfather said. I see my miscalculation. Your entire religious upbringing ... created from the lies of Aharietiam and Honor's own failings ... it pointed you toward this conclusion.
Damnation. The Stormfather wasn't pleased. It suddenly felt terribly unfair. Here he was drinking this awful excuse for wine to follow the ridiculous Codes- he gave every possible outward show of piety- yet it wasn't enough?
"What should I do to serve?" Gavilar said.
You don't understand, the Stormfather said. Those aren't the Words, Gavilar.
"Then what are the storming Words!" he said, slamming the cup down on the table- shattering it, splashing wine across the wall. "You want me to save this planet? Then help me! Tell me what I'm saying wrong!"
It's not about what you are saying.
"But- "
Suddenly the Stormfather wavered. Lightning pulsed through his shimmering form, filling Gavilar's room with an electric glow. Blue frosted the rugs, pure light reflecting in the glass balcony doors.
Then the Stormfather cried out. A sound like a peal of thunder, agonized.
"What?" Gavilar said, backing up. "What happened?"
A Herald ... a Herald has died ... No. I am not ready ... The Oathpact ... No! They mustn't see. They mustn't know ...
"Died?" Gavilar said. "Died. You said they were already dead! You said they were in Damnation!"
The Stormfather rippled, then a face emerged in the shimmering. Two eyes, like holes in a storm, clouds spiraling around them and leading into the depths.
"You lied," Gavilar said. "You lied?"
Oh, Gavilar. There is so much you do not know. So much you assume. And the two never do meet. Like paths to opposing cities.
Those eyes seemed to pull Gavilar forward, to overwhelm him, to consume him. He ... he saw storms, endless storms, and the world was so frail. A tiny speck of blue against an infinite canvas of black.
The Stormfather could lie?
"Restares," Gavilar whispered. "Is he ... actually a Herald?"
Yes.
Gavilar felt cold, as if he were standing in the highstorm, ice seeping in through his skin. Seeking his heart. Those eyes ...
"What are you?" Gavilar whispered, hoarse.
The biggest fool of them all, the Stormfather said. GOODBYE, GAVILAR. I HAVE SEEN A GLIMPSE OF WHAT IS COMING. I WILL NOT PREVENT IT.
"What?" Gavilar demanded. "What is coming?"
YOUR LEGACY.
The door slammed open. Sadeas, his face red from exertion. "Assassin," he said, waving Tearim- in Plate- to tromp in. "Coming this way, killing guards. We need you to put on your armor. Tearim, get it off. We must protect the king."
Gavilar looked at him, stunned.
Then one word cut through.
Assassin.
I've been betrayed, he thought, and found that he was not surprised. One of them had been bound to come for him.
But which one?
"Gavilar!" Sadeas shouted. "We need you in armor! Assassin on the way."
"Tearim can fight him, Torol," Gavilar said. "What is one assassin?"
"This one has killed dozens already," Sadeas said. "I think we should have you in Plate just in case. You could wear mine, but my armorers are still bringing it."
"You brought your armor to the feast?"
"Of course I did," Sadeas said. "I don't trust those Parshendi. You'd do well to emulate me. Trusting too much could get you killed someday."
Screams sounded in the distance. Tearim, loyal as always, began removing the Plate for Gavilar to don.
"Too slow," Sadeas said. "We need to buy time. Give me your robe."
Gavilar hesitated, then met his friend's eyes. "You'd do that?"
"I worked too hard to put you on that throne, Gavilar," Sadeas said, grim. "I'm not going to let that go to waste."
"Thank you," Gavilar said.
Sadeas shrugged, pulling on the robe as Tearim helped Gavilar suit up. Whoever this assassin was, he'd find himself outmatched by a Shardbearer.
Gavilar glanced toward where the Stormfather had been- but the shimmer was gone.
Spren couldn't lie. They couldn't. He'd learned that ... from the Stormfather.
Blood of my fathers, Gavilar thought as the Plate locked onto his legs. What else did it lie to me about?
Gavilar fell.
And he knew, as he fell, that this was it. His ending.
A legacy interrupted. An assassin who moved with an otherworldly grace, stepping on wall and ceiling, commanding Light that bled from the very storms.
Gavilar hit the ground- surrounded by the wreckage of his balcony- and he saw white in a flash. His body didn't hurt. That was an extremely bad sign.
Thaidakar, he thought as a figure rose before him, shadowed in the night air. Only Thaidakar could send an assassin who could do such things.
Gavilar coughed as the figure loomed over him. "I ... expected you ... to come."
The assassin knelt before him, though Gavilar couldn't see anything more than shadows. Then ... the assassin- doing something Gavilar couldn't make out- again glowed like a sphere.
"You can tell ... Thaidakar," Gavilar whispered, "that he's too late ..."
"I don't know who that is," the assassin said, the words barely intelligible. The man held his hand to the side. Summoning a Blade.
This was it. Behind the assassin a halo, a corona of shimmering light. The Stormfather.
I did not cause this, the Stormfather said in his head. I do not know whether that brings you peace in your last moments, Gavilar.
But ...
"Then who ...?" Gavilar forced out. "Restares? Sadeas? I never thought ..."
"My masters are the Parshendi," the assassin said.
Gavilar blinked, focusing on the man once more as his Blade formed. Storms ... that was Jezrien's Honorblade, wasn't it? What was happening?
"The Parshendi? That makes no sense."
This is my failure as much as yours, the Stormfather said. If I try again, I will do it differently. I thought ... your family ...
His family. In that moment, Gavilar saw his legacy crumbling. He was dying.
Storms. He was dying. What did anything matter? He couldn't. He couldn't ...
He was supposed to be eternal ...
I've invited the enemy back, he realized. The end is coming. And my family, my kingdom, will be destroyed, without a way to fight. Unless ...
Hand quivering, he pulled a sphere out of his pocket. The weapon. They needed this. His son ... No, his son could not handle such power ... They needed a warrior. A true warrior. One that Gavilar had been doing his best to suppress, out of a fear he barely dared acknowledge, even as he drew his last ragged breaths.
Dalinar. Storms help them, it came down to Dalinar.
He held the sphere out toward the Stormfather, his vision fuzzing. Thinking ... was ... difficult.
"You must take this," Gavilar whispered to the Stormfather. "They must not get it. Tell ... tell my brother ... he must find the most important words a man can say ..."
No, the Stormfather said, though a hand took the sphere. Not him. I'm sorry, Gavilar. I made that mistake once. I will never trust your family again.
Gavilar gave a whine of pain, not from his body but from his soul. He had failed. He had brought them all to ruin. That, he realized with horror, would be his legacy.
In the end, Gavilar Kholin, heir to the Heralds, died. As all men, ultimately, must.
Alone.

289
book-processing/tests/test_attribution.py Normal file
View File

@ -0,0 +1,289 @@
import json
import os
import re
import unittest
from pathlib import Path
from attribution import (evaluate_attributions, maverick_pronoun_speakers,
quotation_continues, speech_tag_pronouns,
update_entity_relevance)
from label_dialogue import paragraphs_with_word_indexes, read_italic_word_indexes
FIXTURE_ROOT = Path(__file__).parent / "fixtures" / "attribution"
FIXTURES = (
("Prologue", "prologue", "Prologue- To Live"),
("Chapter 1", "chapter-1", "Chapter 1"),
)
class StubMaverick:
"""Return Maverick-shaped clusters for named token occurrences."""
def __init__(self, clusters):
self.clusters = clusters
self.calls = 0
self.inputs = []
def token_count(self, text):
return len(re.findall(r"\w+|[^\w\s]", text)) + 5
def predict(self, text):
self.calls += 1
self.inputs.append(text)
tokens = re.findall(r"\w+|[^\w\s]", text)
offsets = []
for cluster in self.clusters:
offsets.append([
(next(index for index, token in enumerate(tokens)
if token == word and sum(t == word for t in tokens[:index]) == occurrence),) * 2
for word, occurrence in cluster
])
return {"tokens": tokens, "clusters_token_offsets": offsets}
class LateNameMaverick(StubMaverick):
"""Link an earlier pronoun once a later window contains its name."""
def __init__(self):
super().__init__([])
def predict(self, text):
self.calls += 1
self.inputs.append(text)
tokens = re.findall(r"\w+|[^\w\s]", text)
clusters = []
if "Syl" in tokens and "she" in tokens:
clusters.append([(tokens.index("Syl"),) * 2,
(tokens.index("she"),) * 2])
return {"tokens": tokens, "clusters_token_offsets": clusters}
class AttributionEvaluationTests(unittest.TestCase):
def test_maverick_resolves_speech_tag_pronoun(self):
text = 'Syl entered.\n\n"Hello," she said.'
paragraphs = paragraphs_with_word_indexes(text)
model = StubMaverick([[("Syl", 0), ("she", 0)]])
predictions, stats = evaluate_attributions(
paragraphs, {1: "Syl"}, set(), speaker_names=("Syl", "Kaladin"),
maverick_model=model,
)
self.assertEqual(predictions, [None, "Syl"])
self.assertEqual(stats.correct_assignments, 1)
self.assertEqual(model.calls, 1)
def test_maverick_window_keeps_sentence_and_at_most_500_tokens(self):
text = (" ".join(f"word{i}" for i in range(600))
+ '\n\n"Hello," she said.')
paragraphs = paragraphs_with_word_indexes(text)
model = StubMaverick([])
maverick_pronoun_speakers(paragraphs, ("Syl",), model=model)
self.assertEqual(model.calls, 1)
self.assertTrue(model.inputs[0].endswith('"Hello," she said.'))
self.assertLessEqual(model.token_count(model.inputs[0]), 500)
self.assertGreater(model.token_count(model.inputs[0]), 480)
def test_later_window_upgrades_only_other_to_named_entity(self):
text = ('"Hello," she said.\n\nSyl arrived.\n\n'
'"Goodbye," he said.')
paragraphs = paragraphs_with_word_indexes(text)
model = LateNameMaverick()
predictions, _ = evaluate_attributions(
paragraphs, {}, set(), speaker_names=("Syl", "Kaladin"),
maverick_model=model,
)
self.assertEqual(model.calls, 2)
self.assertEqual(predictions[0], "Syl")
def test_quoted_pronoun_and_unresolved_tag_do_not_assign_speaker(self):
text = 'Syl entered.\n\n"She said goodbye," he said.'
paragraphs = paragraphs_with_word_indexes(text)
model = StubMaverick([[("Syl", 0), ("She", 0)]])
mappings = maverick_pronoun_speakers(
paragraphs, ("Syl",), model=model,
)
self.assertEqual(speech_tag_pronouns(paragraphs[1]["text"]), [20])
self.assertEqual(mappings, {})
def test_ambiguous_cluster_and_partly_unresolved_tags_stay_other(self):
text = 'Syl met Kaladin.\n\n"Hello," she said, and he replied.'
paragraphs = paragraphs_with_word_indexes(text)
ambiguous = StubMaverick([[("Syl", 0), ("Kaladin", 0), ("she", 0)]])
partial = StubMaverick([[("Syl", 0), ("she", 0)]])
for model in (ambiguous, partial):
with self.subTest(model=model):
mappings = maverick_pronoun_speakers(
paragraphs, ("Syl", "Kaladin"), model=model,
)
self.assertEqual(mappings, {})
def test_open_quote_continues_across_paragraphs(self):
text = ('Syl stood nearby.\n\n'
'"Listen," she said. "This is not the end.\n\n'
'"Still talking.\n\n'
'"Now done."\n\n'
'"Next?"')
paragraphs = paragraphs_with_word_indexes(text)
model = StubMaverick([[("Syl", 0), ("she", 0)]])
predictions, _ = evaluate_attributions(
paragraphs, {}, set(), speaker_names=("Syl", "Kaladin"),
maverick_model=model,
)
self.assertTrue(quotation_continues(paragraphs[1]["text"],
paragraphs[2]["text"]))
self.assertEqual(predictions, [None, "Syl", "Syl", "Syl", "Kaladin"])
def test_entity_relevance_counts_once_decays_and_stops_at_zero(self):
scores = {"Syl": 0, "Kaladin": 0}
update_entity_relevance(scores, "Syl saw Syl.")
self.assertEqual(scores, {"Syl": 5, "Kaladin": 0})
update_entity_relevance(scores, "Kaladin arrived.")
self.assertEqual(scores, {"Syl": 4, "Kaladin": 5})
for _ in range(10):
update_entity_relevance(scores, "No named characters here.")
self.assertEqual(scores, {"Syl": 0, "Kaladin": 0})
def test_unmarked_dialogue_alternates_by_relevance(self):
text = ('Syl met Kaladin.\n\n'
'"Hi," he said.\n\n'
'"Within reason."\n\n'
'"Whose reason?"')
paragraphs = paragraphs_with_word_indexes(text)
model = StubMaverick([[("Kaladin", 0), ("he", 0)]])
predictions, _ = evaluate_attributions(
paragraphs, {}, set(), speaker_names=("Syl", "Kaladin"),
maverick_model=model,
)
self.assertEqual(predictions, [None, "Kaladin", "Syl", "Kaladin"])
def test_tied_relevance_leaves_unmarked_dialogue_other(self):
text = 'Syl met Kaladin.\n\n"Hello."'
paragraphs = paragraphs_with_word_indexes(text)
predictions, _ = evaluate_attributions(
paragraphs, {}, set(), speaker_names=("Syl", "Kaladin"),
)
self.assertEqual(predictions, [None, "Other"])
def test_always_other_against_chapter_labels(self):
for chapter_name, fixture_name, basename in FIXTURES:
with self.subTest(chapter=chapter_name):
fixture_dir = FIXTURE_ROOT / fixture_name
text = (fixture_dir / f"{basename}.txt").read_bytes().decode("utf-8")
paragraphs = paragraphs_with_word_indexes(text)
labels = json.loads(
(fixture_dir / f"{basename}.json").read_text(encoding="utf-8")
)
italic_word_indexes = read_italic_word_indexes(
fixture_dir / f"{basename}.rsvp", text
)
expected_by_index = {}
for chapter in labels["chapters"]:
for annotation in chapter["paragraphs"]:
index = annotation["paragraph_index"] - 1
self.assertGreaterEqual(index, 0)
self.assertLess(index, len(paragraphs))
self.assertNotIn(index, expected_by_index)
self.assertEqual(
paragraphs[index]["text"].casefold(),
annotation["text"].casefold(),
)
expected_by_index[index] = annotation["entity"]
predictions, stats = evaluate_attributions(
paragraphs, expected_by_index, italic_word_indexes
)
self.assertEqual(len(predictions), stats.total_paragraphs)
self.assertTrue(all(entity in (None, "Other") for entity in predictions))
self.assertEqual(
stats.total_paragraphs,
stats.labeled_paragraphs + stats.unlabeled_paragraphs,
)
self.assertEqual(
stats.labeled_paragraphs,
stats.correct_assignments + stats.incorrect_assignments
+ stats.missed_dialogue + stats.extra_dialogue
+ stats.correct_non_dialogue,
)
self.assertEqual(
stats.expected_dialogue,
stats.correct_assignments + stats.incorrect_assignments
+ stats.missed_dialogue,
)
self.assertEqual(
stats.expected_non_dialogue,
stats.correct_non_dialogue + stats.extra_dialogue,
)
print(f"\nAttribution baseline: {chapter_name} — always Other")
print(f"Paragraphs: {stats.total_paragraphs}")
print(f"Explicitly labeled: {stats.labeled_paragraphs}")
print(f"Unreviewed: {stats.unlabeled_paragraphs}")
print(f"Expected dialogue: {stats.expected_dialogue}")
print(f"Expected non-dialogue: {stats.expected_non_dialogue}")
print(f"Correct assignments: {stats.correct_assignments}")
print(f"Incorrect assignments: {stats.incorrect_assignments}")
print(f"Missed dialogue: {stats.missed_dialogue}")
print(f"Extra dialogue: {stats.extra_dialogue}")
print(f"Correct non-dialogue: {stats.correct_non_dialogue}")
print(f"Predicted dialogue on unreviewed: {stats.predicted_on_unlabeled}")
@unittest.skipUnless(os.environ.get("RUN_MAVERICK_EVAL") == "1",
"set RUN_MAVERICK_EVAL=1 to run the downloaded model")
def test_maverick_against_chapter_labels(self):
for chapter_name, fixture_name, basename in FIXTURES:
with self.subTest(chapter=chapter_name):
fixture_dir = FIXTURE_ROOT / fixture_name
text = (fixture_dir / f"{basename}.txt").read_text(encoding="utf-8")
paragraphs = paragraphs_with_word_indexes(text)
labels = json.loads(
(fixture_dir / f"{basename}.json").read_text(encoding="utf-8")
)
italic_word_indexes = read_italic_word_indexes(
fixture_dir / f"{basename}.rsvp", text
)
expected_by_index = {
annotation["paragraph_index"] - 1: annotation["entity"]
for chapter in labels["chapters"]
for annotation in chapter["paragraphs"]
}
predictions, stats = evaluate_attributions(
paragraphs, expected_by_index, italic_word_indexes,
speaker_names=tuple(labels["entities"].values()),
)
if chapter_name == "Chapter 1":
self.assertEqual(predictions[22], "Syl")
self.assertEqual(predictions[24], "Syl")
named = [
(index + 1, expected_by_index.get(index), prediction)
for index, prediction in enumerate(predictions)
if prediction not in (None, "Other")
]
self.assertTrue(named)
self.assertTrue(all(prediction in labels["entities"].values()
for _, _, prediction in named))
print(f"\nAttribution rules: {chapter_name}")
print(f"Correct assignments: {stats.correct_assignments}")
print(f"Incorrect assignments: {stats.incorrect_assignments}")
errors = [row for row in named if row[1] != row[2]]
print(f"Named predictions: {len(named)}")
print(f"Named errors (first 10): {errors[:10]}")
if __name__ == "__main__":
unittest.main()

74
book-processing/tests/test_epub_to_chapters.py
View File

@ -3,6 +3,8 @@ import unittest
from pathlib import Path
from bs4 import BeautifulSoup
from attribution import dialogue_word_indexes
from label_dialogue import read_italic_word_indexes
from epub_to_chapters import (
SIDECAR_HEADER,
@ -11,12 +13,14 @@ from epub_to_chapters import (
STYLE_BOLD,
STYLE_ITALIC,
STYLE_MARKER,
WORD_COLOR_OTHER,
TIMING_DOUBLE,
TIMING_TRIPLE,
MAX_TOKEN_BYTES,
MAX_SECTION_WORDS,
EnglishVocabulary,
add_unfamiliar_word_timing,
convert_epub,
decode_styled_text,
document_root,
extract_chapter,
@ -32,6 +36,7 @@ from epub_to_chapters import (
substitute_unsupported_characters,
validate_token_lengths,
write_style_sidecar,
style_sidecar_bytes,
write_conversion_report,
)
@ -72,14 +77,35 @@ class DisplayTextNormalizationTests(unittest.TestCase):
self.assertLessEqual(len(name.encode("utf-8")), 200)
def test_section_names_do_not_collide_with_generated_files(self):
used_names = {"toc", "conversion_report"}
def test_section_names_can_match_book_root_metadata(self):
used_names = set()
self.assertEqual(output_section_name("TOC", 1, used_names), "TOC (2)")
self.assertEqual(
output_section_name("CONVERSION_REPORT", 2, used_names),
"CONVERSION_REPORT (2)",
)
self.assertEqual(output_section_name("TOC", 1, used_names), "TOC")
self.assertEqual(output_section_name("CONVERSION_REPORT", 2, used_names),
"CONVERSION_REPORT")
self.assertEqual(output_section_name("TOC", 3, used_names), "TOC (2)")
class BookLayoutTests(unittest.TestCase):
def test_conversion_nests_sections_and_preserves_root_metadata(self):
epub = Path(__file__).resolve().parents[1] / "examples/rsvp-timing-sample.epub"
with tempfile.TemporaryDirectory() as temporary_directory:
output = Path(temporary_directory) / "Sample Book"
convert_epub(epub, output)
titles = output.joinpath("TOC.txt").read_text(encoding="utf-8").splitlines()
self.assertTrue(titles)
self.assertTrue(output.joinpath("CONVERSION_REPORT.txt").is_file())
for title in titles:
self.assertTrue(output.joinpath("content", f"{title}.txt").is_file())
self.assertTrue(output.joinpath("content", f"{title}.rsvp").is_file())
self.assertFalse(output.joinpath(f"{title}.txt").exists())
output.joinpath("NOTES.txt").write_text("Book metadata", encoding="utf-8")
output.joinpath("content", "Old.txt").write_text("Old section")
convert_epub(epub, output, overwrite=True)
self.assertEqual(output.joinpath("NOTES.txt").read_text(), "Book metadata")
self.assertFalse(output.joinpath("content", "Old.txt").exists())
class TokenValidationTests(unittest.TestCase):
@ -182,6 +208,17 @@ class UnsupportedCharacterSubstitutionTests(unittest.TestCase):
class StyleSidecarTests(unittest.TestCase):
def test_marks_dialogue_words_without_marking_surrounding_narration(self):
text = 'Plain "Hello," she said. "Goodbye." Then nothing.'
self.assertEqual(dialogue_word_indexes(text, []), {1, 4})
def test_marks_continued_quotes_and_styled_thoughts(self):
text = '"Hello,\n\n"friend." Outside.\n\nThought silently.'
styled = [(3, STYLE_ITALIC, 0), (4, STYLE_ITALIC, 0)]
self.assertEqual(dialogue_word_indexes(text, styled), {0, 1, 3, 4})
def test_extracts_italic_and_bold_word_styles(self):
soup = BeautifulSoup(
"<p>plain <i>italic</i> <strong>bold</strong> "
@ -225,12 +262,27 @@ class StyleSidecarTests(unittest.TestCase):
self.assertEqual(
data[SIDECAR_HEADER.size:],
bytes([
1, STYLE_ITALIC, 0,
1, STYLE_BOLD, 0,
1, STYLE_ITALIC | STYLE_BOLD, 0,
1, STYLE_ITALIC, 0, 0,
1, STYLE_BOLD, 0, 0,
1, STYLE_ITALIC | STYLE_BOLD, 0, 0,
]),
)
def test_word_color_creates_sparse_records_and_merges_with_style(self):
text = 'Plain "Hello," italic'
records = [(2, STYLE_ITALIC, 0)]
data = style_sidecar_bytes(text, records, {1, 2})
self.assertEqual(SIDECAR_HEADER.unpack_from(data)[4], 2)
self.assertEqual(data[SIDECAR_HEADER.size:], bytes([
1, 0, 0, WORD_COLOR_OTHER,
1, STYLE_ITALIC, 0, WORD_COLOR_OTHER,
]))
with tempfile.TemporaryDirectory() as temporary_directory:
sidecar = Path(temporary_directory) / 'Chapter.rsvp'
sidecar.write_bytes(data)
self.assertEqual(read_italic_word_indexes(sidecar, text), {2})
def test_writes_timing_only_records(self):
text = "Blarenptnip"
@ -241,7 +293,7 @@ class StyleSidecarTests(unittest.TestCase):
self.assertEqual(
data[SIDECAR_HEADER.size:],
bytes([0, 0, TIMING_TRIPLE]),
bytes([0, 0, TIMING_TRIPLE, 0]),
)
def test_extracts_styles_from_css_classes_and_inline_css(self):

337
dialogue-labels.json Normal file
View File

@ -0,0 +1,337 @@
{
"format": "dialogue-attribution-labels-v1",
"entities": {
"1": "Other",
"2": "Kaladin",
"3": "Syl",
"4": "Shallan",
"5": "Pattern"
},
"chapters": [
{
"file": "/home/j2/tds3p-rsvp-reader/Wind and Truth/Chapter 1.txt",
"paragraphs": [
{
"paragraph_index": 1,
"text": "I should have known I was being watched. All my life, the signs were there.",
"entity": "Other"
},
{
"paragraph_index": 2,
"text": "- From Knights of Wind and Truth, page 1",
"entity": null
},
{
"paragraph_index": 5,
"text": "He stood at his window on the first morning of the month. Sunlight streamed into the room around him, wind tickling his hair. He shouldn't have felt good. Yes, he'd helped protect Urithiru- but that victory had come at an agonizing cost. Beyond that, Dalinar had made a deal with the enemy: in just ten days, the champion of Honor and the champion of Odium would decide the fate of all Roshar.",
"entity": null
},
{
"paragraph_index": 7,
"text": "That should have made him an anxious, stewing pot of nerves. Instead he tipped his head back, sun warm on his skin, and acknowledged that while he didn't feel great, someday he would feel great again.",
"entity": null
},
{
"paragraph_index": 13,
"text": "\"Can't you make yourself any size?\" he asked, checking his jacket cuffs.",
"entity": "Kaladin"
},
{
"paragraph_index": 14,
"text": "\"Within reason.\"",
"entity": "Syl"
},
{
"paragraph_index": 15,
"text": "\"Whose reason?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 16,
"text": "\"No idea,\" she said. \"Tried to get as big as a mountain once. It involved lots of grunting and thinking like rocks. Really big rocks. Biggest I could manage was a very small mountain- small enough to fit in this room, with the tip brushing the ceiling.\"",
"entity": "Syl"
},
{
"paragraph_index": 17,
"text": "\"Then you could be tall enough to tower over me,\" he said. \"Why do you usually make yourself shorter?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 18,
"text": "\"It just feels right,\" she said.",
"entity": "Syl"
},
{
"paragraph_index": 19,
"text": "\"That's your explanation for basically everything.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 20,
"text": "\"Yup!\" She poked him. He could barely feel it. Even at this size, she was insubstantial in the Physical Realm. \"Uniform? I thought you weren't going to wear one anymore.\"",
"entity": "Syl"
},
{
"paragraph_index": 21,
"text": "He hesitated, then pulled the jacket down at the bottom to smooth the wrinkles across the sides. \"It just feels right,\" he admitted, meeting her eyes in the mirror.",
"entity": "Kaladin"
},
{
"paragraph_index": 23,
"text": "\"Someone is having a good day,\" she said, poking him again.",
"entity": "Syl"
},
{
"paragraph_index": 24,
"text": "\"Bizarrely,\" Kaladin said. \"Considering.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 25,
"text": "\"At least the war is almost over,\" she said. \"One more contest. Nine days.\"",
"entity": "Syl"
},
{
"paragraph_index": 26,
"text": "True. If Dalinar won, Odium had agreed to withdraw from Alethkar and Herdaz- though he could keep other lands he controlled, like Iri and Jah Keved. If Odium won, they were forced to cede Alethkar to the enemy. Plus there was a greater cost. If Dalinar lost, he had to join Odium, become Fused, and help conquer the cosmere. Kaladin wanted to think that the Radiants wouldn't follow as well, but he wasn't certain. So many people thirsted for war, even without the influence of an Unmade. Storms, he'd felt it too.",
"entity": null
},
{
"paragraph_index": 27,
"text": "\"Syl,\" he said, dropping his smile. \"I'm sure more people are going to die. Perhaps people I care about, but I can't be there to help them. Dalinar will have to choose someone else to be champion and- \"",
"entity": "Kaladin"
},
{
"paragraph_index": 28,
"text": "\"Kaladin Stormblessed,\" she said, rising higher into the air, arms folded. Though she wore a fashionable havah, she left her hair white-blue, flowing free, waving and shifting in the wind. The ... nonexistent wind. \"Don't you dare talk yourself into being miserable.\"",
"entity": "Syl"
},
{
"paragraph_index": 29,
"text": "\"Or what?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 30,
"text": "\"Or I,\" she thundered, \"shall make silly faces at you. As I alone can.\"",
"entity": "Syl"
},
{
"paragraph_index": 31,
"text": "\"They aren't silly,\" he said, shivering.",
"entity": "Kaladin"
},
{
"paragraph_index": 32,
"text": "\"They're hilarious.\"",
"entity": "Syl"
},
{
"paragraph_index": 33,
"text": "\"Last time you made a tentacle come out of your forehead.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 34,
"text": "\"Highbrow comedy.\"",
"entity": "Syl"
},
{
"paragraph_index": 35,
"text": "\"Then it slapped me.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 36,
"text": "\"Punch line. Obviously. All the humans in the world, and I picked the one without a taste for refined humor.\"",
"entity": "Syl"
},
{
"paragraph_index": 38,
"text": "\"It does feel warm,\" he said, \"to have finally figured a few things out. To let go of the weight and step out from the shadow. I know darkness will return, but I think ... I think I'll be able to remember better than before.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 39,
"text": "\"Remember what?\"",
"entity": "Syl"
},
{
"paragraph_index": 40,
"text": "He Lashed himself upward, floating until he was eye level with her. \"That days like this exist too.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 42,
"text": "\"I wish I could show Teft,\" Kaladin said. \"I feel his loss like a hole in my own flesh, Syl.\"",
"entity": "Kaladin"
},
{
"paragraph_index": 43,
"text": "\"I know,\" she said softly.",
"entity": "Syl"
},
{
"paragraph_index": 49,
"text": "And he would remember.",
"entity": null
},
{
"paragraph_index": 51,
"text": "Kaladin ...",
"entity": "Other"
},
{
"paragraph_index": 53,
"text": "\"Syl,\" Kaladin said, \"did you say my name?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 54,
"text": "\"Huh?\" she said, opening her eyes.",
"entity": "Syl"
},
{
"paragraph_index": 55,
"text": "Kaladin ...",
"entity": "Other"
},
{
"paragraph_index": 57,
"text": "I need your help. I'm so sorry ... to ask more of you ...",
"entity": "Other"
},
{
"paragraph_index": 58,
"text": "\"Tell me you hear that,\" Kaladin said to Syl.",
"entity": "Kaladin"
},
{
"paragraph_index": 59,
"text": "\"I feel ...\" She cocked her head. \"I feel something. On the wind.\"",
"entity": "Syl"
},
{
"paragraph_index": 60,
"text": "\"It's speaking to me,\" he said, one hand to his head.",
"entity": "Kaladin"
},
{
"paragraph_index": 61,
"text": "A storm is coming, Kaladin, the wind whispered. The worst storm ... I'm sorry ...",
"entity": "Other"
},
{
"paragraph_index": 63,
"text": "\"What did you hear?\" Syl asked.",
"entity": "Syl"
},
{
"paragraph_index": 64,
"text": "\"A warning,\" he said, frowning. \"Syl, is the wind ... alive?\"",
"entity": "Kaladin"
},
{
"paragraph_index": 65,
"text": "\"Everything is alive.\"",
"entity": "Syl"
},
{
"paragraph_index": 69,
"text": "Indeed, life was largely about perspective.",
"entity": null
},
{
"paragraph_index": 70,
"text": "Like this strange structure: a hollow, rectangular block hundreds of feet tall, dominating Shadesmar's landscape. People- spren- lived along the inside walls, walking up and down them, ignoring conventions of gravity. Looking down along one of the inside walls could be stomach-churning unless you changed your perspective. Unless you convinced yourself that walking up and down that wall was normal. Whether a person was strong or not wasn't usually subject to debate, yet if gravity could be a matter of opinion ...",
"entity": null
},
{
"paragraph_index": 73,
"text": "She'd bonded two. One during her childhood. One as an adult. She'd hurt the first, and had suppressed the memory.",
"entity": null
},
{
"paragraph_index": 75,
"text": "Nearby, Pattern's head pulsed to a vibrant rhythm- always moving, always forming some new geometric display. Comparing the two broke Shallan's heart. She had done this to Testament by rejecting the bond after using her Shardblade to kill her mother.",
"entity": null
},
{
"paragraph_index": 78,
"text": "\"Why?\" Shallan asked. \"Why don't you hate me?\"",
"entity": "Shallan"
},
{
"paragraph_index": 79,
"text": "Pattern rested his hand on Shallan's shoulder. \"We both knew the danger, the sacrifice, in bonding to humans again.\"",
"entity": "Pattern"
},
{
"paragraph_index": 80,
"text": "\"I hurt her.\"",
"entity": "Shallan"
},
{
"paragraph_index": 81,
"text": "\"Yet here you are,\" Pattern said. \"Able to stand tall. Able to control the Surges. Able to protect this world.\"",
"entity": "Pattern"
},
{
"paragraph_index": 82,
"text": "\"She should hate me,\" Shallan whispered. \"But there is no vitriol in the way she holds my hand. No judgment in the way she remains with us.\"",
"entity": "Shallan"
},
{
"paragraph_index": 83,
"text": "\"Because the sacrifice was worth something, Shallan,\" Pattern said, uncharacteristically reserved. \"It worked. In the end you recovered, did better. I am still here. And remarkably, I am not even a little bit dead! I do not think you will kill me at all, Shallan! I am happy about that.\"",
"entity": "Pattern"
},
{
"paragraph_index": 84,
"text": "\"Can I heal her?\" Shallan asked. \"Maybe if I ... if I bond her again?\"",
"entity": "Shallan"
},
{
"paragraph_index": 85,
"text": "\"I think, after talking to Kelek ...\" Pattern said. \"I think you are still bonded to her.\"",
"entity": "Pattern"
},
{
"paragraph_index": 86,
"text": "\"But ...\" Shallan glanced over her shoulder at him. \"I broke the bond. That did this.\"",
"entity": "Shallan"
},
{
"paragraph_index": 87,
"text": "\"Some breaks are messy,\" Pattern said. \"A slice with a sharpened knife is clean; a slice with a dull one is ragged. Your break, done by a child without full Intent, is ragged. In some ways that makes it worse, but it does mean that some Connection between you two persists.\"",
"entity": "Pattern"
},
{
"paragraph_index": 88,
"text": "\"So ...\"",
"entity": "Shallan"
},
{
"paragraph_index": 89,
"text": "\"So no,\" Pattern said. \"I do not think that merely saying Words once more would heal her.\" His head pattern spun a little more slowly, as if he were contemplating something profound. \"These numbers are ... perplexing, Shallan. Strangely irrational, in a sequence I do not understand. I mean ... I mean that we are walking on unfamiliar ground. A better metaphor for you. Yes. Unfamiliar ground. In the deep past, deadeyes did not exist.\"",
"entity": "Pattern"
},
{
"paragraph_index": 91,
"text": "The result had been the deadeyes. The explanation might lie with Kelek, the very person Shallan had been sent to Lasting Integrity to kill. She squeezed Testament's hand. \"I'm going to help you,\" Shallan whispered. \"Whatever it takes.\"",
"entity": "Shallan"
},
{
"paragraph_index": 93,
"text": "\"Thank you,\" Shallan said. \"For coming to me when I was young. Thank you for protecting me. I still do not remember it all, but thank you.\"",
"entity": "Shallan"
},
{
"paragraph_index": 95,
"text": "\"Rest now,\" Shallan said, wiping her eyes and standing. \"I'm going to figure this out.\"",
"entity": "Shallan"
}
]
}
]
}

80
docs/history/PLAN.md Normal file
View File

@ -0,0 +1,80 @@
# Historical text coloring implementation plan
This proposal predates the current v3 sidecar and dialogue coloring
implementation. Its entity format, palette, and rollout order are not the
current design. See the root README for implemented behavior and TODO.md for
remaining work.
## Goal and order
Validate color rendering on the T-Display S3 Pro before changing EPUB
attribution. First add firmware support for sparse v3 entity metadata and a
card-ready lorem ipsum book that repeats the full palette. Flash the board and
inspect the colors, gray, white, word transitions, and reading speed. Only then
build the converter's dialogue and thought attribution.
The revised palette has nine hues: 0, 30, 60, 120, 180, 225, 270, 300, and
330 degrees, all at HSL saturation 100% and lightness 75%. HSL (0, 0%, 75%) is
the gray for Other and uncolored entities. Ordinary words remain white. Hue
360 is equivalent to 0, so the map uses 0.
## Phase 1: firmware and hardware book
1. Accept v3 RSVP sidecars while preserving v1/v2 behavior. V3 keeps the
existing magic, source fingerprint, and record count, then adds a 64-bit
entity-catalog fingerprint. Each sparse record retains its word-index delta,
style flags, and timing modifier. The high bit of the style byte means an
entity ID follows as an unsigned varint. The entity ID may be **0**, which
is the real category **Other**. An absent high bit means no entity. A word
with no style, timing, or entity assignment has no sidecar record.
2. Load an editable `ENTITY_COLORS.tsv` once per book. It maps entity IDs to
hues, with an entity-catalog fingerprint to reject a stale map. ID 0 always
uses gray; named entities with no mapped hue also use gray. A missing or
invalid map leaves assigned words gray. Multiple entities may share a hue;
the map has a 64 KiB file-size limit but no entity-count limit. V1/v2 books
remain readable.
3. Add a saved Colors on/off setting and a sixth menu item. Off is the default
and renders all words white as before. On colors only entity-tagged reading
words; chapter cards, menus, and ordinary words remain white. Convert hues
to RGB565 when loading the book and keep the palette in memory, with no SD
reads during word advances.
4. Include a card-ready `Color Cycle` book containing only repeated lorem ipsum
words. In each 55-word cycle, each of the nine hues colors five consecutive
words, followed by five words with entity ID 0 (gray) and five with no
entity annotation (white). Repeat the cycle many times so playback and
manual navigation expose rendering artifacts.
5. Build the firmware and verify the fixture's word order, v3 fingerprint,
map fingerprint, and color annotations. Hardware acceptance requires the
user to observe all eleven five-word color blocks across repeated cycles, clean
transitions, and the persisted toggle after a restart.
## Phase 2: attribution rules and statistics
1. Preserve quote boundaries, EPUB style cues, and final word indexes through
normalization and character substitution. Identify conventional paired
quotation marks and only clearly marked or context-supported thought spans;
generic italics alone do not imply thought.
2. Attribute explicit named speech/thought tags as **clear**. Use surrounding
context and a bounded two-speaker exchange for **reasonable guesses**,
including multi-paragraph continuations. Explicit pronoun-only tags such as
he, she, and it map to **Other**. Unresolved dialogue/thought also maps to
Other; text that is not dialogue/thought remains unassigned.
3. Process the example book iteratively and report aggregate word and span
counts for clear, guessed, and unattributable cases. Refine the rules until
the user likes that distribution. Do not present passages, per-span review,
or a manual speaker/alias correction workflow in the terminal UI.
4. Detect canonical character names and safe aliases automatically. Color name
mentions outside dialogue by their entity; inside dialogue, the speaker's
color takes precedence. Ambiguous name matches remain unassigned.
## Phase 3: terminal color assignment and converter output
After attribution, list entities by descending dialogue word count in the
terminal, without excerpts. Let the user assign any of the nine hues to each
entity; the same hue may be assigned to multiple entities. The user may stop at
any point, leaving remaining named entities gray. Other remains gray. Display
ANSI true-color swatches when supported, with numeric hue labels as the
fallback. Write the editable per-book map and v3 sidecars, keeping metadata
sparse and leaving the source text unchanged.
Update the README and cover token alignment, attribution statistics, sidecar
compatibility, map failures, size limits, and on-device performance.

5
firmware/include/Board.h
View File

@ -10,10 +10,11 @@ class Board {
bool beginSd();
void setReadingContext(const String& bookTitle, const String& chapterTitle,
uint32_t wordCount, uint8_t progressPercent);
void drawWord(const String& word, uint16_t wpm, uint8_t styleFlags = 0);
void drawWord(const String& word, uint16_t wpm, uint8_t styleFlags = 0,
uint8_t wordColorFlag = 0);
void drawChapterTitle(const String& chapter, uint16_t wpm);
void drawTheEnd(uint16_t wpm);
void drawMenu(uint8_t selectedOption);
void drawMenu(uint8_t selectedOption, bool colorsEnabled);
void drawPicker(const String& title, const String& value);
void drawFontStylePicker(uint8_t style, uint8_t size);
void drawFontSizePicker(uint8_t style, uint8_t size);

3
firmware/include/Library.h
View File

@ -5,7 +5,8 @@
class Library {
public:
bool loadLastOrFirst(String& path, String& contents) const;
bool loadLastOrFirst(String& path, String& contents,
bool& resumedSavedChapter) const;
bool load(const String& path, String& contents) const;
bool loadSidecar(const String& chapterPath, std::vector<uint8_t>& bytes) const;
bool selectBook(const String& currentPath, int direction, String& chapterPath) const;

29
firmware/include/Palette.h Normal file
View File

@ -0,0 +1,29 @@
#pragma once
#include <stddef.h>
#include <stdint.h>
namespace palette {
constexpr uint16_t rgb565(uint8_t red, uint8_t green, uint8_t blue) {
return (static_cast<uint16_t>(red & 0xF8) << 8) |
(static_cast<uint16_t>(green & 0xFC) << 3) | (blue >> 3);
}
constexpr uint16_t kOtherGrey = rgb565(176, 176, 176);
constexpr uint16_t kHues[] = {0, 30, 60, 120, 180, 225, 270, 300, 330};
constexpr uint16_t kColors[] = {
rgb565(255, 128, 128), rgb565(255, 191, 128), rgb565(255, 255, 128),
rgb565(128, 255, 128), rgb565(128, 255, 255), rgb565(128, 159, 255),
rgb565(191, 128, 255), rgb565(255, 128, 255), rgb565(255, 128, 191),
};
inline int indexForHue(uint16_t hue) {
for (size_t index = 0; index < sizeof(kHues) / sizeof(kHues[0]); ++index) {
if (kHues[index] == hue) return static_cast<int>(index);
}
return -1;
}
} // namespace palette

3
firmware/include/Reader.h
View File

@ -14,7 +14,9 @@ struct StyleToken {
uint32_t index;
uint8_t styleFlags;
uint8_t timingModifier;
uint8_t wordColorFlag;
};
static_assert(sizeof(StyleToken) == 8, "Word metadata must remain memory-budgeted");
class Reader {
public:
@ -30,6 +32,7 @@ class Reader {
uint8_t progressPercent() const;
uint32_t delayMs(uint16_t wpm) const;
uint8_t currentStyleFlags() const;
uint8_t currentColorFlag() const;
bool sidecarValid() const;
private:

1
firmware/include/Settings.h
View File

@ -9,6 +9,7 @@ struct SavedState {
uint8_t brightness = 5;
uint8_t fontStyle = 0;
uint8_t fontSize = 20;
bool colorsEnabled = true;
};
class Settings {

19
firmware/src/Board.cpp
View File

@ -7,6 +7,7 @@
#include <XPowersLib.h>
#include "Config.h"
#include "Palette.h"
namespace {
PowersSY6970 powerMonitor;
@ -241,7 +242,8 @@ void Board::drawFooter() {
displayedProgressPercent_ = progressPercent_;
}
void Board::drawWord(const String& word, uint16_t wpm, uint8_t styleFlags) {
void Board::drawWord(const String& word, uint16_t wpm, uint8_t styleFlags,
uint8_t wordColorFlag) {
if (display_ == nullptr) return;
setLandscape();
@ -270,7 +272,7 @@ void Board::drawWord(const String& word, uint16_t wpm, uint8_t styleFlags) {
renderedSize -= config::kFontSizeStep;
setReaderFont(fontStyle_, renderedSize, styleFlags);
}
display_->setTextColor(TFT_WHITE);
display_->setTextColor(wordColorFlag == 1 ? palette::kOtherGrey : TFT_WHITE);
display_->setTextDatum(MC_DATUM);
const int16_t centerX = display_->width() / 2;
@ -308,7 +310,7 @@ void Board::drawChapterTitle(const String& chapter, uint16_t wpm) {
void Board::drawTheEnd(uint16_t wpm) { drawWord("The End", wpm); }
void Board::drawMenu(uint8_t selectedOption) {
void Board::drawMenu(uint8_t selectedOption, bool colorsEnabled) {
if (display_ == nullptr) return;
if (landscape_) {
@ -325,13 +327,14 @@ void Board::drawMenu(uint8_t selectedOption) {
display_->setTextDatum(TC_DATUM);
display_->drawString("MENU", display_->width() / 2, 18);
constexpr int16_t kRowHeight = 58;
constexpr int16_t kFirstRowY = 60;
constexpr int16_t kRowSpacing = 68;
constexpr int16_t kRowHeight = 48;
constexpr int16_t kFirstRowY = 54;
constexpr int16_t kRowSpacing = 56;
const char* labels[] = {"Back", "Book selection", "Chapter selection",
"Brightness", "Font"};
"Brightness", "Font",
colorsEnabled ? "Colors: On" : "Colors: Off"};
display_->setTextDatum(ML_DATUM);
for (uint8_t index = 0; index < 5; ++index) {
for (uint8_t index = 0; index < 6; ++index) {
const int16_t y = kFirstRowY + index * kRowSpacing;
if (index == selectedOption) {
display_->fillRoundRect(10, y, display_->width() - 20, kRowHeight, 5, TFT_WHITE);

63
firmware/src/Library.cpp
View File

@ -5,27 +5,52 @@
#include <vector>
#include "Config.h"
#include "Settings.h"
namespace {
constexpr char kContentDirectory[] = "content";
bool isLegacyChapterPath(const String& path) {
const String prefix = String(config::kBooksDirectory) + "/";
if (!path.startsWith(prefix) || !path.endsWith(".txt")) return false;
const String relativePath = path.substring(prefix.length());
const int separator = relativePath.indexOf('/');
return separator > 0 && separator == relativePath.lastIndexOf('/');
}
bool isChapterPath(const String& path) {
const String prefix = String(config::kBooksDirectory) + "/";
if (!path.startsWith(prefix) || !path.endsWith(".txt")) return false;
const String relativePath = path.substring(prefix.length());
const int separator = relativePath.indexOf('/');
return separator > 0 && separator == relativePath.lastIndexOf('/');
const int bookSeparator = relativePath.indexOf('/');
if (bookSeparator <= 0) return false;
const String contentPath = relativePath.substring(bookSeparator + 1);
const String contentPrefix = String(kContentDirectory) + "/";
return contentPath.startsWith(contentPrefix) &&
contentPath.length() > contentPrefix.length() + 4 &&
contentPath.lastIndexOf('/') == contentPrefix.length() - 1;
}
String migratedChapterPath(const String& path) {
if (!isLegacyChapterPath(path)) return String();
const int separator = path.lastIndexOf('/');
return path.substring(0, separator) + "/" + kContentDirectory +
path.substring(separator);
}
} // namespace
bool Library::loadLastOrFirst(String& path, String& contents) const {
Settings settings;
settings.begin();
const SavedState state = settings.load();
bool Library::loadLastOrFirst(String& path, String& contents,
bool& resumedSavedChapter) const {
resumedSavedChapter = false;
// Loading the saved file verifies that it still exists. Avoid walking every
// TOC entry at boot; that turns a resume into one SD-card open per chapter.
if (isChapterPath(state.bookPath) && loadFile(state.bookPath, contents)) {
path = state.bookPath;
if (isChapterPath(path) && loadFile(path, contents)) {
resumedSavedChapter = true;
return true;
}
const String migratedPath = migratedChapterPath(path);
if (isChapterPath(migratedPath) && loadFile(migratedPath, contents)) {
path = migratedPath;
resumedSavedChapter = true;
return true;
}
return firstBook(path) && loadFile(path, contents);
@ -72,6 +97,10 @@ String parentPath(const String& path) {
return slash > 0 ? path.substring(0, slash) : String();
}
String bookDirectory(const String& chapterPath) {
return parentPath(parentPath(chapterPath));
}
String childPath(const String& parent, const String& name) {
return name.startsWith("/") ? name : parent + "/" + name;
}
@ -111,7 +140,7 @@ bool listChapters(const String& path, std::vector<String>& chapters,
// TOC.txt is the source of truth for both chapter order and membership.
// Opening every listed file here makes navigation scale with book length.
chapters.push_back(path + "/" + title + ".txt");
chapters.push_back(path + "/" + kContentDirectory + "/" + title + ".txt");
titles.push_back(title);
}
toc.close();
@ -142,19 +171,19 @@ bool Library::selectBook(const String& currentPath, int direction,
if (!ensureBooks()) return false;
String selectedDirectory;
if (!cycle(books_, parentPath(currentPath), direction, selectedDirectory)) return false;
if (!cycle(books_, bookDirectory(currentPath), direction, selectedDirectory)) return false;
return firstChapter(selectedDirectory, chapterPath);
}
bool Library::selectChapter(const String& currentPath, int direction,
String& chapterPath) const {
const String directoryPath = parentPath(currentPath);
const String directoryPath = bookDirectory(currentPath);
if (directoryPath.isEmpty() || !ensureChapters(directoryPath)) return false;
return cycle(chapters_, currentPath, direction, chapterPath);
}
bool Library::previousChapter(const String& currentPath, String& chapterPath) const {
const String directoryPath = parentPath(currentPath);
const String directoryPath = bookDirectory(currentPath);
if (directoryPath.isEmpty() || !ensureChapters(directoryPath)) return false;
for (size_t index = 1; index < chapters_.size(); ++index) {
@ -167,7 +196,7 @@ bool Library::previousChapter(const String& currentPath, String& chapterPath) co
}
bool Library::nextChapter(const String& currentPath, String& chapterPath) const {
const String directoryPath = parentPath(currentPath);
const String directoryPath = bookDirectory(currentPath);
if (directoryPath.isEmpty() || !ensureChapters(directoryPath)) return false;
for (size_t index = 0; index + 1 < chapters_.size(); ++index) {
@ -180,11 +209,11 @@ bool Library::nextChapter(const String& currentPath, String& chapterPath) const
}
String Library::bookName(const String& chapterPath) const {
return baseName(parentPath(chapterPath));
return baseName(bookDirectory(chapterPath));
}
String Library::chapterName(const String& chapterPath) const {
const String directoryPath = parentPath(chapterPath);
const String directoryPath = bookDirectory(chapterPath);
if (chapterDirectory_ == directoryPath && !chapters_.empty()) {
for (size_t index = 0; index < chapters_.size(); ++index) {
if (chapters_[index] == chapterPath) return chapterTitles_[index];

21
firmware/src/Reader.cpp
View File

@ -13,8 +13,10 @@ constexpr uint8_t kStyleBold = 0x02;
constexpr uint8_t kKnownStyleFlags = kStyleItalic | kStyleBold;
constexpr uint8_t kSidecarVersion1 = 1;
constexpr uint8_t kSidecarVersion2 = 2;
constexpr uint8_t kSidecarVersion3 = 3;
constexpr uint8_t kTimingDouble = 2;
constexpr uint8_t kTimingTriple = 3;
constexpr uint8_t kWordColorOther = 1;
constexpr size_t kSidecarHeaderBytes = 18;
uint32_t readUint32Le(const std::vector<uint8_t>& bytes, size_t offset) {
@ -177,7 +179,8 @@ bool Reader::loadSidecar(const std::vector<uint8_t>& sidecar) {
if (sidecar.size() < kSidecarHeaderBytes ||
sidecar[0] != 'R' || sidecar[1] != 'S' ||
sidecar[2] != 'V' || sidecar[3] != 'P' ||
(sidecar[4] != kSidecarVersion1 && sidecar[4] != kSidecarVersion2) ||
(sidecar[4] != kSidecarVersion1 && sidecar[4] != kSidecarVersion2 &&
sidecar[4] != kSidecarVersion3) ||
sidecar[5] != 0 ||
readUint64Le(sidecar, 6) != sourceFingerprint_) {
return false;
@ -194,7 +197,9 @@ bool Reader::loadSidecar(const std::vector<uint8_t>& sidecar) {
for (uint32_t recordNumber = 0; recordNumber < recordCount; ++recordNumber) {
uint32_t delta = 0;
if (!readVarint(sidecar, offset, delta) || offset + 2 > sidecar.size()) {
const size_t recordBytes = sidecarVersion == kSidecarVersion3 ? 3 : 2;
if (!readVarint(sidecar, offset, delta) ||
offset + recordBytes > sidecar.size()) {
return false;
}
@ -205,15 +210,18 @@ bool Reader::loadSidecar(const std::vector<uint8_t>& sidecar) {
}
const uint8_t styleFlags = sidecar[offset++];
const uint8_t timingModifier = sidecar[offset++];
const uint8_t wordColorFlag = sidecarVersion == kSidecarVersion3
? sidecar[offset++] : 0;
if (wordIndex >= tokens_.size() ||
(styleFlags & ~kKnownStyleFlags) != 0 ||
(timingModifier != 0 && timingModifier != kTimingDouble &&
timingModifier != kTimingTriple) ||
(sidecarVersion == kSidecarVersion1 && timingModifier != 0) ||
(styleFlags == 0 && timingModifier == 0)) {
wordColorFlag > kWordColorOther ||
(styleFlags == 0 && timingModifier == 0 && wordColorFlag == 0)) {
return false;
}
records.push_back({wordIndex, styleFlags, timingModifier});
records.push_back({wordIndex, styleFlags, timingModifier, wordColorFlag});
previousIndex = wordIndex;
}
@ -322,6 +330,11 @@ uint8_t Reader::currentStyleFlags() const {
return token ? token->styleFlags : 0;
}
uint8_t Reader::currentColorFlag() const {
const StyleToken* token = currentStyleToken();
return token ? token->wordColorFlag : 0;
}
bool Reader::sidecarValid() const { return sidecarValid_; }
uint32_t Reader::delayMs(uint16_t wpm) const {

5
firmware/src/Settings.cpp
View File

@ -18,6 +18,7 @@ SavedState Settings::load() const {
state.brightness = preferences.getUChar("brightness", 5);
state.fontStyle = preferences.getUChar("font_style", 0);
state.fontSize = preferences.getUChar("font_size", 20);
state.colorsEnabled = preferences.getBool("colors", true);
preferences.end();
return state;
}
@ -45,5 +46,9 @@ void Settings::save(const SavedState& state) const {
if (!preferences.isKey("font_size") || preferences.getUChar("font_size") != state.fontSize) {
preferences.putUChar("font_size", state.fontSize);
}
if (!preferences.isKey("colors") ||
preferences.getBool("colors") != state.colorsEnabled) {
preferences.putBool("colors", state.colorsEnabled);
}
preferences.end();
}

29
firmware/src/main.cpp
View File

@ -97,7 +97,8 @@ void setReadingContext(uint8_t progressPercent) {
void showCurrentWord() {
setReadingContext(reader.progressPercent());
board.drawWord(reader.currentWord(), state.wpm, reader.currentStyleFlags());
board.drawWord(reader.currentWord(), state.wpm, reader.currentStyleFlags(),
state.colorsEnabled ? reader.currentColorFlag() : 0);
}
void wakeDisplay();
@ -172,7 +173,7 @@ void openMenu() {
menuSelection = 0;
screen = Screen::kMenu;
wakeDisplay();
board.drawMenu(menuSelection);
board.drawMenu(menuSelection, state.colorsEnabled);
}
void openPicker(Screen picker) {
@ -332,6 +333,11 @@ void confirmMenuSelection() {
case 2: openPicker(Screen::kChapterPicker); break;
case 3: openPicker(Screen::kBrightnessPicker); break;
case 4: openPicker(Screen::kFontStylePicker); break;
case 5:
state.colorsEnabled = !state.colorsEnabled;
persistState();
board.drawMenu(menuSelection, state.colorsEnabled);
break;
}
return;
}
@ -390,8 +396,8 @@ void handleInput() {
if (released(previous)) {
wakeDisplay();
if (screen == Screen::kMenu) {
menuSelection = (menuSelection + 4) % 5;
board.drawMenu(menuSelection);
menuSelection = (menuSelection + 5) % 6;
board.drawMenu(menuSelection, state.colorsEnabled);
return;
}
if (screen != Screen::kReader) {
@ -410,8 +416,8 @@ void handleInput() {
if (released(next)) {
wakeDisplay();
if (screen == Screen::kMenu) {
menuSelection = (menuSelection + 1) % 5;
board.drawMenu(menuSelection);
menuSelection = (menuSelection + 1) % 6;
board.drawMenu(menuSelection, state.colorsEnabled);
return;
}
if (screen != Screen::kReader) {
@ -474,7 +480,10 @@ void setup() {
String book;
std::vector<uint8_t> sidecar;
if (!library.loadLastOrFirst(state.bookPath, book)) {
const String savedBookPath = state.bookPath;
const uint32_t savedWord = state.wordIndex;
bool resumedSavedChapter = false;
if (!library.loadLastOrFirst(state.bookPath, book, resumedSavedChapter)) {
board.drawWord("NO BOOKS", state.wpm);
while (true) delay(1000);
}
@ -483,10 +492,10 @@ void setup() {
board.drawWord("NO BOOKS", state.wpm);
while (true) delay(1000);
}
const uint32_t savedWord = state.wordIndex;
reader.restoreAtParagraphStart(savedWord);
reader.restoreAtParagraphStart(resumedSavedChapter ? savedWord : 0);
state.wordIndex = reader.currentIndex();
if (sanitizedSettings || state.wordIndex != savedWord) persistState();
if (sanitizedSettings || state.bookPath != savedBookPath ||
state.wordIndex != savedWord) persistState();
readerView = ReaderView::kChapterTitle;
showReaderScreen();
lastInteractionMs = millis();