How tshatrov/ichiran works
Main Components
• Core Processing Engine: Several Lisp source files (e.g., characters.lisp, romanize.lisp, cli.lisp, ichiran.lisp, and many dict-*.lisp files) form the heart of the system. These contain the algorithms for romanization, segmentation, and dictionary management.
• Dictionary Management: Multiple modules (like dict-counters.lisp, dict-custom.lisp, dict-errata.lisp, dict-fix.lisp, dict-grammar.lisp, dict-load.lisp, etc.) handle importing, processing, and managing the JMdictDB data and other dictionary sources.
• Data Sources: The “data” directory contains CSV and XML files (e.g., romaji-map.csv, extra.xml, gyoseiku.csv, jichitai.csv) that supply essential mapping and supplementary information.
• Database Layer: The project utilizes a Postgres database (as indicated by docker/postgres-dockerfile, docker/postgres-initdb scripts, and the instructions to initialize or recreate the database). This is the backend for storing and querying dictionary data.
• Dockerized Deployment: The Docker-related files (e.g., .dockerignore, docker-compose.yml, docker scripts) show that the system is containerized. There are specific containers for SBCL (the Common Lisp runtime), for running the CLI, and for managing the Postgres database.
• Testing & Maintenance: Files such as tests.lisp and maintenance commands in the README indicate automated test suites and DB initialization/maintenance utilities.
• (Future) Web Interface: Though still experimental and under development, the README refers to a web interface currently accessible at ichi.moe.
Relationships and Interactions
• The CLI and (future) web interface act as front-ends that receive user input.
• These front-ends interact with the core processing engine, which processes text (e.g., romanizing and segmenting input) by calling various utility and dictionary management modules.
• The engine consults the database for extra information (word meanings, readings, etc.) via connection modules (e.g., conn.lisp) and potentially via designated functions in settings.lisp.
• Docker orchestrates these components by running separate containers (e.g., SBCL container for the Lisp code and a Postgres container for the database). The docker-compose setup defines and links these containers together so that inter-component communication is seamless.
• Maintenance scripts and test suites serve as auxiliary elements ensuring that the database is properly initialized and that the functionality of the text-processing modules is validated.
Architectural Patterns & Design Principles
• Separation of Concerns: The system is modular with clearly delineated responsibilities (text processing vs. dictionary management vs. database operations vs. container orchestration).
• Loose Coupling & Clear Dependencies: The core processing components depend on specific modules (e.g., dictionary loading) and external data (CSV, XML, and database), but they are orchestrated through well-defined ASDF systems.
• Containerization: Docker file configuration reinforces a micro-service-like environment where each service (database and the Lisp runtime) remains isolated yet interconnected.
• Extensibility: With a clean division between core algorithm modules and I/O components (CLI, web interface), extending the web interface or adding new language processing features should be straightforward.