Chessboard photo to FEN

Task 1: no learned models

Everything downstream depends on a clean perspective warp, so the pipeline spends most of its effort on the board rather than on the pieces. Every figure below follows one photo, G006_IMG048, from the raw frame through to the boxes.

Finding the board. Two detectors run independently on the grayscale image. One blurs and thresholds for white; the other runs Canny edge detection directly. Each keeps the contour whose convex hull has the largest area, then absorbs its neighbours: any contour lying within a distance threshold of that hull is merged in, the hull is recomputed, and the sweep repeats until nothing else merges.

The two results are then intersected, and cv2.approxPolyDP simplifies the intersection to a four-vertex polygon, whose corners sort into board order by coordinate. That intersection is where the robustness comes from. White thresholding breaks under strong lighting, Canny latches onto nearby pieces or the floor, and the two rarely fail on the same image, so their overlap is far steadier than either alone.

Orientation comes from template matching. A crop of the knight logo printed on the board is rotated to each of the four possible corner orientations and matched against the warped image. The best-scoring rotation names the alignment.

The grid. CLAHE for local contrast, Gaussian blur, Canny, then dilation to close the broken edges. A Hough transform pulls out straight lines, which are filtered by angle and clustered to merge duplicates. Lines still go missing, so the gaps are filled by measuring the median spacing between the surviving lines and inferring where the absent ones belong. Intersecting the horizontal and vertical sets gives candidate points; keeping the 81 closest to the board centre, subject to a minimum separation, gives the 9×9 lattice. Without that separation constraint the selection collapses into a ring of points around the centre.

Occupancy. Each square is cropped and handed to cv2.grabCut with a hint mask of two concentric ellipses, sitting slightly above the square's centre because a chess piece is tall and narrow. The inner ellipse is labelled GC_FGD, the outer GC_PR_FGD. A square counts as occupied only if the resulting foreground ratio falls inside a band. GrabCut tends to either flood the whole square or return nothing when it fails, and both failure modes land outside that band.

Bounding boxes were attempted twice. The first version thresholds in HSV and cleans up with erosion and dilation. It works for white pieces, whose hue separates cleanly, but produces many false positives and negatives on black ones; a distance-transform threshold and watershed improve it without fixing it. The second version instead starts from the GrabCut mask and iterates watershed on it, letting the mask grow past the square boundary and rebuilding the hint mask each round by eroding for sure foreground and dilating for sure background. Boxes are then read off by inverse-warping the contours back to the original image and fitting boxes there, rather than warping the boxes themselves.

The second method scores slightly better and is the one used, but it leans on an accurate warp and degrades badly when corner detection goes wrong. A bad corner fit is detectable, so those images fall back to the HSV method, which never needed the warp in the first place.

On the 50-image test subset, occupancy reaches a mean F1 of 0.9039 (median 0.9387). Bounding boxes, scored by F1 at an IoU threshold of 0.5, reach 0.7336 (median 0.7560). The gap between the two is the expected one: deciding whether a square holds a piece is a much easier question than drawing a box around that piece under overlap and perspective distortion.

Task 2: counting the pieces

The second task predicts one number, the count of pieces on the board, as a regression scored by MAE.

The baseline is a ResNet50 with pre-trained weights, fine-tuned end to end for 30 epochs, MSE loss, AdamW at 1e-4 with weight decay 0.01, and a sigmoid head rescaled into the legal range with out = 30·σ(x) + 2. Images were resized to 256 and cropped to 224. That setup reaches 0.73 validation MAE, and everything after it is a greedy sweep: try one change, keep it if it helps, carry it into the next experiment.

Random cropping was deliberately left out of the augmentation set, since a crop that clips the board edge removes pieces and silently corrupts the label.

Optuna settled on L1 loss, a learning rate of 5.26e-4, weight decay 0.0326 and cosine annealing. That configuration was then trained for 30, 100 and 200 epochs, with the 100 and 200 epoch runs saving their best checkpoint by rounded MAE rather than plain MAE. The 200-epoch model is the best of them, at 0.0027 rounded MAE on validation and 0.0056 on test.

A board holds a whole number of pieces, and the raw predictions land close enough to the truth that rounding almost always recovers the exact count: on validation, a plain MAE of 0.0665 becomes 0.0027. Selecting checkpoints on the rounded metric is also the likely reason validation and test disagree slightly.

One result from Task 3 cuts against the premise of this one. The best YOLO detector, run over the full validation set at a confidence threshold of 0.5 and asked only to count its own detections, reaches 0.0233 validation and 0.0202 test MAE with no rounding at all, better than the dedicated regressor's unrounded numbers. The report puts this down to the detector's size and more robust training. The specialised model only wins once rounding is applied.

Task 3: the digital twin

Piece detection was never the bottleneck. YOLOv8n was the baseline, trained 100 epochs at batch size 64 with images resized to 960px and otherwise default settings, and reached mAP50 0.99 and mAP50-95 0.86. YOLOv8x and YOLO11s were trained afterwards; YOLOv8x came out on top at mAP50 0.995 and mAP50-95 0.875.

Mapping those boxes onto squares is where it got hard. An end-to-end approach following Masouris et al., the authors of the ChessReD dataset, classified each square directly through a fully connected head under cross-entropy loss. It overfit badly and performed poorly on anything beyond common openings.

So detection and board geometry were split apart: YOLO finds the pieces, a separate step finds the board, and each detection collapses to a single anchor point: horizontally centred, 90% of the way down the box, so it sits at the piece's base rather than on its body. Warping that point onto the rectified grid names a square, and that assignment is enough to write out a FEN string.

The pipeline is scored by edit distance between the expanded FEN and the ground truth: the minimum number of single-character insertions, deletions and substitutions needed to turn one into the other, which is a direct count of misplaced, wrong and missing pieces.

The first version of this pipeline used Task 1's classical corner detection, and across the whole of ChessReD2k it averaged 1.59 edits per board. Storing the mismatched strings and reading through them showed where the edits were coming from, and the size of the edit distance turned out to identify the cause:

The distribution is bimodal rather than spread out: a board is usually either exactly right or badly wrong, and the badly wrong ones are mostly orientation, where the corners are found well enough but applied the wrong way round, rotating the entire position.

That made corner detection the real task. Five approaches were built and compared.

Corner quality is measured as mean Euclidean distance (MED) between predicted and true corner coordinates, normalised, where 0% would be perfect.

Approach A is the Task 1 pipeline reused, and it is the one that motivated the rest: it relies on Canny edges, polygon approximation and template matching, and it struggles under occlusion and difficult lighting.

Augmentation made the difference for the CNN regressor. C1 and C2 are the same idea: a CNN predicting four corner coordinates directly. C1 froze a ResNet50 backbone, trained a few added layers with MSE and Adam behind a sigmoid output, and was still unreliable after 150 epochs at 13.19% test MED. C2 switched to EfficientNetV2-S, unfroze every layer and added manual augmentation. Most augmentation libraries transform the image and leave the labels behind, which is useless when the label is a set of coordinates, so this needed tv_tensors.KeyPoints from a pre-release torchvision 0.23 to carry the corners through each rotation and perspective shift. That reached 0.45%.

YOLO11s-pose found the board but not its corners. Trained 100 epochs at batch size 64 and 640px, it located the board almost perfectly, with mAP50 of 1.0 on the board's bounding box, but its predicted keypoints rarely coincided with the true corners.

The segmentation route hit an annotation problem. D segments the board with a U-Net, reads the corners off the mask, and hands orientation to a separate ResNet18 classifying 0°, 90°, 180° or 270°. A mask says nothing about which corner is which, so the classifier is doing the work. But the corner annotations mark only the board's internal corners and crop out the knight logo, the one feature that settles orientation. The fix was to feed the classifier a deliberately wider crop with enough margin to keep the logo in frame.

The heatmap U-Net scored best and still mislabels corners. Predicting a probability distribution per corner rather than a coordinate gave well-defined peaks sitting on the true corners, at 0.31% test MED. But it sometimes swaps the heatmaps for the top-left and bottom-right corners. It learns to find corner-like features without reliably assigning the right label from the board's global orientation, which is the same failure the edit-distance analysis had already pointed at.

The final pipeline pairs YOLOv8x with the heatmap U-Net: find the corners, warp the board, drop each detection to its anchor point, read off the square, emit FEN.

Run over the whole of ChessReD2k, that pipeline averages 0.0197 edits per board, roughly two misplaced pieces per hundred boards, against 1.59 for the version that used classical corners. Almost all of the eighty-fold gap comes from the catastrophic cases disappearing: better corners mean fewer boards where the warp is wrong and nothing lands on the right square. The two written reports stop at 1.59 and never re-measure, so that final figure comes from the delivery notebook rather than from either PDF.

The set with annotated corners is small, which caps how well any of these generalise across varied conditions. Augmentation was only ever applied in the simplest of the corner methods, where it clearly helped, and the labelling problem calls for an explicit orientation module rather than asking the corner model to resolve it implicitly.