Prepare a text corpus.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Start with a collection you can explain

A corpus is a collection of text used for a purpose. Before training a library assistant, ask where each notice came from, whether you may use it, when it was written, and whether it contains information that should not enter training. A license is permission under specified terms; it does not establish accuracy, privacy or fitness for your task. A document’s provenance records its origin and transformations.

Our downloadable collection is explicitly fictional and licensed for reuse. Some documents are copied from existing Learn AI fixtures; others are declared formatting changes, a paraphrase or new authored examples. It lets us inspect every decision, but cannot establish coverage of real communities or languages. For a real project, record collection dates, source identifiers, permissions and exclusions before building the dataset; keep that record with a version and file hash. A hash identifies bytes, not the truth of their provenance.

Separate documents before making training chunks

Suppose a notice says the library closes at 17:00. One website repeats it exactly; another changes the case and spaces. If one copy trains the model and another evaluates it, a good score may reflect remembered text. Deduplication removes repeated documents under a declared comparison rule. Exact matching misses formatting changes. Normalized matching may ignore case and spacing, but still misses a paraphrase. It can also erase meaningful distinctions: case matters in some code and names.

Keep known related documents in the same source group, then assign that group to a partition. In this lab the relationships and group assignments are authored and visible. In real data you must investigate them; a URL is not automatically an independent source. Group splitting does not guarantee that all hidden relationships are known. For time-sensitive deployment, a chronological split may also be needed.

Only after assignment should you cut long documents into training windows. Keep every window from a source in its source’s partition. Splitting windows independently can put almost identical overlapping passages on both sides. Save document boundaries so the model does not silently learn that the end of one unrelated notice leads into the next.

A filter changes whose text remains

A filter that rejects non-ASCII characters deletes café and 雪 along with anything you intended to remove. Short-document filters can remove useful notices. There is no universally correct threshold: inspect examples and measure retention by relevant content, language and source groups before deciding. Do not equate a smaller vocabulary or simpler spelling with better data.

The lab counts length in Unicode code points, not bytes or guaranteed visible characters. Filtering removes entire documents. Normalization is used only to compare duplicate keys; the retained text is not rewritten. An empty corpus needs a repair, not a success badge.

Fit the vocabulary on the permitted partition

Fitting a tokenizer learns its vocabulary or merge rules from text. Fit it on training documents, then encode development documents using that fixed tokenizer. If development text contributes new vocabulary entries, its unknown-token rate no longer measures coverage on unseen fitting text. When choosing among filters or tokenizers using development feedback, label those experiments repeated development. Reserve a different evaluation set for a later frozen decision.

Here the word tokenizer preserves case, whitespace and punctuation. One uppercase word can be unknown despite its lowercase counterpart being present. Unknown rate is the fraction of encoded development tokens absent from the fitted vocabulary. It does not measure whether a model predicts, reasons or serves users well. If there is no fitted vocabulary or no development text, that rate is undefined.

Mixing changes exposure, not independent evidence

Imagine 60 training tokens from notices and 20 from community documents. The original community share is 20 out of 80, or 25%. Giving each community document three times the exposure contributes 60 community tokens against 60 notice tokens:

The numbers 1 and 3 are exposure weights; 60 and 20 are token counts before weighting. The numerator counts weighted community tokens. The denominator counts all weighted tokens. The share rises, but the number of unique source documents does not. Longer documents also contribute more tokens, so equal document weights do not guarantee equal token shares.

Our lab computes those counts directly. It does not train a language model, change the word vocabulary when exposure weights change, or claim that more repetition improves outcomes. A trained system needs a matched evaluation before making that claim.

Apply the preparation decisions

For a guided experiment, export two preparation choices and explain one leakage repair and one filtering cost using retained document IDs. For implementation practice, download the JSON, reproduce the retained IDs and weighted token shares, and test an exact duplicate, a paraphrase and an empty result. Preserve source metadata and do not mix development text into fitting. For a decision brief, propose an intended audience, name an unacceptable exclusion and specify the additional real data and frozen evaluation needed before launch.

These are practice artifacts. A completed note is not independent implementation evidence or a human review. Keep the original source files and transformations so another person can reproduce the choices.