# subaru-pfs/spt_target_uploader architecture

> Architecture diagram of the GitHub repository subaru-pfs/spt_target_uploader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/subaru-pfs/spt_target_uploader
- Repository: https://github.com/subaru-pfs/spt_target_uploader

## Overview

##### Main Components
- Web User Interface: Built using templates (see the “templates” directory with files like admin_login.html.jinja2) which render the login and other user screens.
- Command Line Interface (CLI): Located under src/pfs_target_uploader/cli (for example, cli_main.py), this serves as the entry point to launch and interact with the web service.
- Application Core: The project’s core logic is split into modules such as pn_app.py and pn_admin.py, which likely handle the main operations and administrative functions.
- Utility Modules: Under the “utils” directory, there are helper modules (e.g., checker.py, db.py, io.py, mail.py, ppp.py, session.py) that support various back-end functionalities including file IO, database operations, email handling, and session management.
- UI Widgets: The “widgets” directory contains components to present interactive elements (like DatePicker, FileInput, and Status widgets) that are potentially embedded within the web interface.
- Data Persistence and Storage: The project stores uploaded files in a “data” directory with a structured layout (organized by year/month and timestamp-upload_id), and uses an SQLite database (UPLOADID_DB) for duplication checks.
- External Services: Although not directly part of the code, the application interacts with external resources such as an SMTP server for email, and possibly external clustering libraries (e.g., HDBSCAN/DBSCAN).

##### Relationships and Data Flow
- The CLI (pfs-uploader-cli) is used to start the app, which spins up a server listening on a specified port (e.g., http://localhost:5008/).
- User Actions: Users interact with the web UI for operations like uploading target files.
- On file upload, the system processes the file using the application core, calling utility functions (e.g., db.py for duplication checking, io.py for reading/writing, and mail.py for notifications).
- The widgets aid in dynamically presenting status information and upload results in the web interface.
- Processed data is then archived into the “data” directory in a structured format, and critical metadata (upload IDs) is recorded in a local SQLite database.
- Configuration of the system (such as setting OUTPUT_DIR, log levels, and clustering algorithms) is performed via the .env.shared file, which the relevant modules refer to at runtime.

##### Architectural Patterns & Technologies
- Modular Architecture: The project cleanly separates components (CLI, core logic, utilities, widgets, templates) to ensure maintainability and easy extensibility.
- CLI-driven Web Application: Using the CLI to initiate the web server is a design choice that makes it easily automatable.
- Template Engine for Views: Jinja2 (or a similar templating mechanism) is used for dynamic HTML rendering.
- File System-Based Data Persistence: Uploaded files are organized and stored on disk, while a lightweight SQLite database ensures no duplication in submissions.
- Environment-based Configuration: Reading configuration parameters from .env files allows flexible deployment and testing.

##### Final Recommendations
- Major components and their interrelationships are captured.
- Incorporate any relevant external dependencies (such as SMTP servers and clustering libraries).
- Avoid overcomplicating; focus on clear, modular separation of responsibilities while including as many unique components from the file tree as possible.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction
    U["User Browser"]:::user

    %% Frontend / User Interface
    subgraph "Frontend UI"
        T1["Admin Login Template"]:::frontend
        T2["Basic Login Template"]:::frontend
        W["UI Widgets"]:::frontend
    end

    %% CLI and Application Core
    subgraph "Backend Processing"
        C["CLI Launcher"]:::cli
        A1["pn_app"]:::core
        A2["pn_admin"]:::core
    end

    %% Utility Modules Layer
    UTL["Utility Modules"]:::utility

    %% Data Persistence Layer
    subgraph "Data Storage"
        D["Data Directory"]:::data
        DB["SQLite Database"]:::data
    end

    %% External Services Layer
    subgraph "External Services"
        SMTP["SMTP Server"]:::external
        CL["Clustering Libraries"]:::external
    end

    %% Configuration
    CONFIG["Environment Config (.env)"]:::config

    %% Connections

    %% CLI launches the server which serves the Frontend UI
    C -->|"launches"| T1
    C --> T2
    C --> W

    %% User interacts with UI
    U -->|"accesses"| T1
    U --> T2
    U --> W

    %% Frontend UI submits data to Application Core
    T1 -->|"triggers processing"| A1
    T2 --> A1
    W  --> A1

    %% Application Core components interaction
    A1 -->|"admin tasks"| A2

    %% Application Core calls Utility Modules
    A1 --> UTL
    A2 --> UTL

    %% Utility Modules persist data and perform operations
    UTL -->|"writes files"| D
    UTL -->|"records metadata"| DB
    UTL -->|"sends email"| SMTP
    UTL -->|"uses clustering"| CL

    %% Configuration feeds into key components
    CONFIG --> C
    CONFIG --> A1
    CONFIG --> A2
    CONFIG --> UTL

    %% Click Events
    click T1 "https://github.com/subaru-pfs/spt_target_uploader/blob/main/templates/admin_login.html.jinja2"
    click T2 "https://github.com/subaru-pfs/spt_target_uploader/blob/main/templates/basic_login_admin.html"
    click W "https://github.com/subaru-pfs/spt_target_uploader/tree/main/src/pfs_target_uploader/widgets/"
    click C "https://github.com/subaru-pfs/spt_target_uploader/blob/main/src/pfs_target_uploader/cli/cli_main.py"
    click A1 "https://github.com/subaru-pfs/spt_target_uploader/blob/main/src/pfs_target_uploader/pn_app.py"
    click A2 "https://github.com/subaru-pfs/spt_target_uploader/blob/main/src/pfs_target_uploader/pn_admin.py"
    click UTL "https://github.com/subaru-pfs/spt_target_uploader/tree/main/src/pfs_target_uploader/utils/"

    %% Styles
    classDef user fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef frontend fill:#ccf2ff,stroke:#333,stroke-width:2px;
    classDef cli fill:#ffd1a4,stroke:#333,stroke-width:2px;
    classDef core fill:#ffb3b3,stroke:#333,stroke-width:2px;
    classDef utility fill:#fff2cc,stroke:#333,stroke-width:2px;
    classDef data fill:#d0f0c0,stroke:#333,stroke-width:2px;
    classDef external fill:#ffe6cc,stroke:#333,stroke-width:2px;
    classDef config fill:#e6e6e6,stroke:#333,stroke-width:2px;
```
