Teach useful invariances.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

First decide what the label means

Imagine recognizing whether a small bar is vertical or horizontal. Moving the complete bar a little to the right changes its pixels but not its orientation. Rotating it a quarter turn changes the answer. Cropping away its arms leaves a dot with no visible orientation. The transformation alone cannot tell you whether keeping the label is justified: you need the task definition and the transformed evidence.

Data augmentation creates transformed training examples. Keeping the same target asserts that the transformation preserves the answer. Invariance means the desired answer stays unchanged under a specified transformation. It is a task-specific property to test, not a promise that all transformations help.

A rotation might preserve a label such as “contains a bar” while changing “vertical versus horizontal.” A crop might preserve a species label if the identifying animal remains visible, but not if only background remains. Our lab uses exact synthetic bars to make these distinctions inspectable; it does not test recognition of animals or photographs.

Trace the pixels and their target

The two original training sources are centered three-pixel bars. Each has a five-by-five brightness grid. Vertical is label 0 and horizontal is label 1. A valid shift moves the center to one of nine positions where the complete bar fits. The brightness option uses both 1 and 0.5. Pixels not on the bar stay zero. Neither changes the orientation in this controlled task.

The two fault switches are intentional counterexamples. A 90-degree rotation changes the orientation while the training data incorrectly retain the old label. The destructive crop keeps only the center pixel; vertical and horizontal sources then become indistinguishable at a given center and brightness. The optimizer receives the supplied target, even when it is wrong or no longer inferable.

Inspect the original and transformed grids before training. State what information remains. Do not infer target validity from a low loss or a high development score.

Train a small actual network

The model has 25 brightness inputs, eight hidden units using tanh, and one sigmoid output. It has 217 trainable parameters: eight groups of 25 input weights plus one bias (208), then eight output weights plus one bias (9). Unlike the preceding sliding filter, these hidden units have separate weights at each input location; this is a dense network, not a convolutional encoder.

Each hidden unit multiplies pixel values by its weights, adds them and a bias, then applies tanh, which maps the sum into the range −1 to 1. The output combines the eight hidden values into a logit, a real-valued score. Sigmoid maps that score to a number between 0 and 1, interpreted here as the predicted probability of horizontal. The rule selects horizontal at probability 0.5 or above; this interpretation does not establish calibration.

For a correct horizontal target, predicting 0.8 gives a loss of −ln(0.8), about 0.223. Predicting only 0.2 gives −ln(0.2), about 1.609. The natural logarithm, ln, makes confident wrong predictions more costly. For a vertical target, use the predicted vertical probability, 1 minus the horizontal probability.

Here ℓ is one image’s loss, y is its supplied target (0 or 1), and p is the model’s horizontal probability. Training minimizes the mean of these losses. The implementation calculates an equivalent stable expression from logits to avoid rounding a probability to zero before taking a logarithm. Every update computes all gradients from the same preceding weights, then changes all weights together.

Nearby practice: a vertical target receives horizontal probability 0.2. Which probability belongs inside its loss?

Compare interventions without changing everything

Capture the centered baseline, then enable shifts and restart from the same deterministic weights. Use the same learning rate and number of updates. The model should be judged on development cases as well as training fit. Compare counts and losses, and inspect actual mistakes.

The baseline contains two images. Adding shifts gives eighteen. At 1,000 full-batch updates, those runs visit 2,000 versus 18,000 training examples. Equal update counts therefore do not mean equal work. The lab reports visits and measured worker time separately; neither is an estimate of production cost.

The eighteen fixed development bars have brightness 0.75 and all fitting positions. Their arrays do not occur in training, but they share the same bar generator. They are visible after every run, so repeated comparisons make them development evidence. They are not an untouched reserved test, independent original sources, or evidence about complex real images. A future perception project needs broader data, a declared split and independent evaluation.

Read a failure rather than hiding it

Wrong rotations create identical patterns with opposing labels. Predicting 0.5 gives loss ln(2), about 0.693, on balanced conflicting targets. No deterministic image-only model can give different answers to exactly identical pixels unless additional distinguishing information is supplied.

Cropping reveals a different trap: development accuracy can stay high on uncropped bars despite ambiguous training dots. That result does not vindicate the crop. It shows that this development population does not test the destructive condition. Inspect both the training assumptions and the evaluation coverage.

For a guided report, explain one transformed image and compare two matched runs. For implementation, reproduce forward computation, check gradients and distinguish augmented variants from independent sources. For a decision brief, state an intended task, defend allowed transformations and specify a missing evaluation population. A completed practice report is separate from independent project assessment and human review.