How dhruv16s/chesssense works
1. What this repository is
ChessSense is a local Python computer-vision pipeline, not a web service. Its job is to take a chessboard image captured from the screen, detect the board and pieces, convert the position to FEN, and query a bundled Stockfish engine for a best move.
2. Top-level orchestration
app.py is the entry point and likely coordinates the whole flow end-to-end: acquire image/frame, detect the board, infer piece locations, build FEN, and request an engine move recommendation. That makes it the main system boundary for the application runtime.
3. Input domain and assumptions
The project is tuned for a specific chess.com visual style: Neo pieces on the green board configuration. This matters architecturally because the recognition pipeline depends on reference templates and calibrated thresholds rather than being generic across arbitrary chess UIs.
4. Board detection subsystem
detect_board.py appears to locate the chessboard within a larger screenshot using contouring and other image-processing steps. board_lines.py then detects or reconstructs the board’s internal grid so the board can be split into 64 cells. This is the geometric foundation for everything downstream.
5. Piece recognition subsystem
recognize_pieces_positions.py is the core classifier/matcher for identifying which piece occupies each square. It likely compares board-cell crops against template images rather than using a learned model. The pieces/ and resized_pieces/ directories provide the reference images used for matching.
6. Template and asset storage
board_templates/ holds board reference images, while pieces/black_pieces and pieces/white_pieces contain piece templates under different background conditions. resized_pieces/ is a processed copy sized for matching. These folders are data assets, not executable logic, and they form the recognition model’s “training/reference set.”
7. Color-threshold calibration
The thresholds/ directory exists to separate white and black pieces using color statistics. thresholds/define_thresholds.py and the plots in that folder indicate an offline calibration step used to derive decision boundaries for piece-color classification. This is important because the board/piece colors are part of the recognition signal.
8. Image preprocessing utilities
image_augmentation.py and image_resizing.py prepare the template data into usable forms for matching. This is an offline preprocessing layer that supports robustness by creating consistent template sizes and possibly more varied samples.
9. Position encoding
fen_notation.py converts the recognized piece layout into a FEN string. This is a key boundary: it transforms vision output into a chess-engine-friendly representation, decoupling image analysis from chess logic.
10. Move suggestion engine
get_best_move.py uses the generated FEN to query stockfish.exe, which is bundled in the repo. This is a separate external computation boundary: the computer vision system hands off board state to a deterministic engine process and receives a best move back.
11. Data flow through the system
The runtime flow is: screen/image input → board detection → board segmentation into squares → template/threshold-based piece recognition → FEN generation → Stockfish evaluation → recommended move. Intermediate outputs may be saved for inspection, especially in the board-detection stage.
12. Operational footprint
requirements.txt defines the Python environment, while the bundled stockfish.exe makes the engine dependency local and self-contained. Overall, the repository is a single-machine computer-vision pipeline with offline-prepared assets and a lightweight runtime orchestrator.
Main entry points: recognize_pieces_positions.py Recognize pieces · detect_board.py Detect board · image_augmentation.py Augment images · image_resizing.py Resize images
Components 16
Runtime pipeline
Live image-to-move processing path
- App · Main orchestrator for capture, vision, FEN, and engine query
app.py - Detect board · Finds the chessboard in a screenshot
detect_board.py - Board grid · Reconstructs board lines and 64-square layout
board_lines.py - Recognize pieces · Matches square crops to piece templates and colors
recognize_pieces_positions.py - FEN encode · Converts recognized board state into FEN
fen_notation.py - Best move · Sends FEN to Stockfish and returns a recommendation
get_best_move.py - Stockfish · Bundled chess engine used for evaluation
stockfish.exe
Reference assets
Template images and generated match data
- Board templates · Reference board images for board detection
board_templates - Black pieces · Black piece reference images
pieces/black_pieces - White pieces · White piece reference images
pieces/white_pieces - Resized black · Processed black piece templates for matching
resized_pieces/black_pieces - Resized white · Processed white piece templates for matching
resized_pieces/white_pieces - Augment images · Creates varied training/reference images
image_augmentation.py - Resize images · Normalizes template dimensions for matching
image_resizing.py
Calibration
Offline threshold tuning and analysis
- Thresholds · Stored calibration artifacts for color-based separation
thresholds - Define thresholds · Offline tool for deriving piece-color thresholds
thresholds/define_thresholds.py
Connections 17
- App → Detect board: capture frame · Start vision pipeline on an input image
- Detect board → Board templates: use templates · Matches board appearance against board references
- Detect board → Board grid: board region · Pass detected board for grid reconstruction
- Board grid → Recognize pieces: 64 squares · Produce square crops for piece identification
- Recognize pieces → Black pieces: match black · Compare crops against black piece templates
- Recognize pieces → White pieces: match white · Compare crops against white piece templates
- Recognize pieces → Resized black: aligned refs · Use resized black templates during matching
- Recognize pieces → Resized white: aligned refs · Use resized white templates during matching
- Recognize pieces → Thresholds: color cues · Apply calibrated thresholds to separate piece colors
- Recognize pieces → FEN encode: piece map · Send recognized board layout for encoding
- FEN encode → Best move: fen · Convert board state into engine input
- Best move → Stockfish: query · Invoke bundled engine process
- Augment images → Black pieces: generate · Preprocess source templates into augmented references
- Augment images → White pieces: generate · Preprocess source templates into augmented references
- Resize images → Resized black: build · Create matching-ready black templates
- Resize images → Resized white: build · Create matching-ready white templates
- Define thresholds → Thresholds: calibrate · Derive and store threshold values