How subaru-pfs/spt_target_uploader works
Main Components
• Web User Interface: Built using templates (see the “templates” directory with files like admin_login.html.jinja2) which render the login and other user screens.
• Command Line Interface (CLI): Located under src/pfs_target_uploader/cli (for example, cli_main.py), this serves as the entry point to launch and interact with the web service.
• Application Core: The project’s core logic is split into modules such as pn_app.py and pn_admin.py, which likely handle the main operations and administrative functions.
• Utility Modules: Under the “utils” directory, there are helper modules (e.g., checker.py, db.py, io.py, mail.py, ppp.py, session.py) that support various back-end functionalities including file IO, database operations, email handling, and session management.
• UI Widgets: The “widgets” directory contains components to present interactive elements (like DatePicker, FileInput, and Status widgets) that are potentially embedded within the web interface.
• Data Persistence and Storage: The project stores uploaded files in a “data” directory with a structured layout (organized by year/month and timestamp-upload_id), and uses an SQLite database (UPLOADID_DB) for duplication checks.
• External Services: Although not directly part of the code, the application interacts with external resources such as an SMTP server for email, and possibly external clustering libraries (e.g., HDBSCAN/DBSCAN).
Relationships and Data Flow
• The CLI (pfs-uploader-cli) is used to start the app, which spins up a server listening on a specified port (e.g., http://localhost:5008/).
• User Actions: Users interact with the web UI for operations like uploading target files.
• On file upload, the system processes the file using the application core, calling utility functions (e.g., db.py for duplication checking, io.py for reading/writing, and mail.py for notifications).
• The widgets aid in dynamically presenting status information and upload results in the web interface.
• Processed data is then archived into the “data” directory in a structured format, and critical metadata (upload IDs) is recorded in a local SQLite database.
• Configuration of the system (such as setting OUTPUT_DIR, log levels, and clustering algorithms) is performed via the .env.shared file, which the relevant modules refer to at runtime.
Architectural Patterns & Technologies
• Modular Architecture: The project cleanly separates components (CLI, core logic, utilities, widgets, templates) to ensure maintainability and easy extensibility.
• CLI-driven Web Application: Using the CLI to initiate the web server is a design choice that makes it easily automatable.
• Template Engine for Views: Jinja2 (or a similar templating mechanism) is used for dynamic HTML rendering.
• File System-Based Data Persistence: Uploaded files are organized and stored on disk, while a lightweight SQLite database ensures no duplication in submissions.
• Environment-based Configuration: Reading configuration parameters from .env files allows flexible deployment and testing.
Final Recommendations
• Major components and their interrelationships are captured.
• Incorporate any relevant external dependencies (such as SMTP servers and clustering libraries).
• Avoid overcomplicating; focus on clear, modular separation of responsibilities while including as many unique components from the file tree as possible.