# super50nic/project_elf_owl architecture

> Architecture diagram of the GitHub repository super50nic/project_elf_owl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/super50nic/project_elf_owl
- Repository: https://github.com/super50nic/project_elf_owl

## Overview

##### Main high-level components
a. External Vulnerability Data Sources: NVD, GitHub, GitLab, OSS-Fuzz, Snyk, OSV, package repos, etc.
b. Ingestion & Refinement Pipeline: “vulnerabilities” module and “elfowl” backend code.
c. Storage Layer: PostgreSQL database.
d. Web Application & API: Django app under “vulnerablecode” (with dashboard UI in “elfowl/dashboard” and public JSON API).
e. CLI & Aggregator: “vulntotal” command-line interface and validator.
f. Infrastructure & DevOps: nginx reverse proxy, Docker/Docker-Compose, GitHub Actions, Nix shells.

##### Relationships & data flows
- Importers (pull raw advisory feeds)
- Improvers (normalize, map to Package-URLs, enrich data)
- Pipeline to PostgreSQL: “INSERT / UPDATE vulnerability & package tables.”
- Django Web App (vulnerablecode module) consuming the same DB: serving UI (dashboard templates, static files) and REST API (DRF with spectacular docs).  Illustrate browser/users → nginx → Gunicorn/WSGI → Django.
- “vulntotal” CLI as a separate client that can query multiple upstream datasources AND optionally the VulnerableCode API.

##### Architectural patterns
- Pipeline pattern for data ingestion: import → improve → store.
- Monolithic codebase but logically split into reusable modules (vulnerabilities, elfowl, vulnerablecode, vulntotal).
- Django MVC for the web tier: models ↔ views ↔ templates + static assets.
- Management commands as batch jobs.

##### Technology stacks
- “nginx” as reverse proxy and static file server.
- “GitHub Actions” and “Nix / Makefiles” for CI/dev environments.

- vulnerabilities.importers.* (list the major importers like NginxImporter, GitHubImporter, NVDImporter)
- vulnerabilities.improvers.* (default, valid_versions, vulnerability_status)
- elfowl/backend/main.py (entry point), code_analyzer.py, openai_handler.py, llm_trainer.py
- Django apps: vulnerablecode/context_processors, vulnerablecode/urls.py, vulnerablecode/views, elfowl/dashboard
- API docs: DRF Spectacular under vulnerablecode/static/js/swagger_ui.html
- CLI: vulntotal/vulntotal_cli.py, vulntotal/datasources/*

##### Putting it all together
- nginx
- Django Web App container (Gunicorn + vulnerablecode + elfowl/dashboard)
- elfowl Processing Engine container
- PostgreSQL container
Outside, place “vulnerabilities” batch workers triggered via “manage.py import/improve” (these can run in separate containers or Kubernetes jobs).  Also outside, place “vulntotal CLI” and various External Sources.

## Diagram (Mermaid)

```mermaid
graph TB
    %% VulnerableCode System Container
    subgraph "VulnerableCode System (Docker)"
        subgraph "nginx (Reverse Proxy)"
            Nginx["nginx"]:::infra
        end
        subgraph "Web App Container (Gunicorn + Django)"
            Gunicorn["Gunicorn/WSGI"]:::web
            Django["Django App (vulnerablecode)"]:::web
            Dashboard["elfowl/dashboard"]:::web
        end
        subgraph "elfowl Engine Container"
            Main["main.py"]:::engine
            Analyzer["code_analyzer.py"]:::engine
            LLM["llm_trainer.py"]:::engine
            OpenAI["openai_handler.py"]:::engine
            Thread["thread_handler.py"]:::engine
            Secret["secret_finder.py"]:::engine
            DBHandler["database_handler.py"]:::engine
        end
        PostgreSQL["PostgreSQL"]:::db

        Nginx -->|HTTP| Gunicorn
        Gunicorn -->|WSGI| Django
        Django -->|serves UI/API| Dashboard
        Gunicorn -->|SQL queries| PostgreSQL
        Main -->|SQL queries| PostgreSQL
        Main --> Analyzer --> LLM --> OpenAI --> Thread --> Secret --> DBHandler
    end

    %% Vulnerabilities Batch Workers
    subgraph "Vulnerabilities Batch Workers"
        subgraph vulnerabilities
            ImportRunner["import_runner.py"]:::ingestion
            ImporterBase["importer.py"]:::ingestion
            ImportersDir["importers/"]:::ingestion
            ImproveRunner["improve_runner.py"]:::ingestion
            ImproverBase["improver.py"]:::ingestion
            ImproversDir["improvers/"]:::ingestion

            ImportRunner -->|step1| ImportersDir
            ImportersDir -->|raw data| ImproverBase
            ImproveRunner -->|step2| ImproversDir
            ImproversDir -->|INSERT/UPDATE| PostgreSQL
        end
    end

    %% External Sources
    subgraph "External Vulnerability Data Sources"
        NVD["NVD"]:::external
        GitHub["GitHub"]:::external
        GitLab["GitLab"]:::external
        OSV["OSV"]:::external
        Snyk["Snyk"]:::external

        NVD -->|importers| ImportersDir
        GitHub -->|importers| ImportersDir
        GitLab -->|importers| ImportersDir
        OSV -->|importers| ImportersDir
        Snyk -->|importers| ImportersDir
    end

    %% CLI Client
    VulnTotal["vulntotal CLI"]:::cli
    VulnTotal -->|CLI args| NVD
    VulnTotal -->|HTTP/JSON API| Django

    %% CI/CD & Dev
    subgraph "CI/CD & Dev Environments"
        GH["GitHub Actions"]:::infra
        Nix["Nix Shell"]:::infra
        Make["Makefile"]:::infra
    end

    %% Click Events
    click ImportersDir "https://github.com/super50nic/project_elf_owl/tree/main/vulnerabilities/importers/"
    click ImportRunner "https://github.com/super50nic/project_elf_owl/blob/main/vulnerabilities/import_runner.py"
    click ImporterBase "https://github.com/super50nic/project_elf_owl/blob/main/vulnerabilities/importer.py"
    click ImproveRunner "https://github.com/super50nic/project_elf_owl/blob/main/vulnerabilities/improve_runner.py"
    click ImproverBase "https://github.com/super50nic/project_elf_owl/blob/main/vulnerabilities/improver.py"
    click ImproversDir "https://github.com/super50nic/project_elf_owl/tree/main/vulnerabilities/improvers/"

    click PostgreSQL "https://github.com/super50nic/project_elf_owl/blob/main/docker-compose.yml"
    click Django "https://github.com/super50nic/project_elf_owl/blob/main/vulnerablecode/settings.py"

    click Main "https://github.com/super50nic/project_elf_owl/blob/main/elfowl/backend/main.py"
    click Analyzer "https://github.com/super50nic/project_elf_owl/blob/main/elfowl/backend/code_analyzer.py"
    click LLM "https://github.com/super50nic/project_elf_owl/blob/main/elfowl/backend/llm_trainer.py"
    click OpenAI "https://github.com/super50nic/project_elf_owl/blob/main/elfowl/backend/openai_handler.py"
    click Thread "https://github.com/super50nic/project_elf_owl/blob/main/elfowl/backend/thread_handler.py"
    click Secret "https://github.com/super50nic/project_elf_owl/blob/main/elfowl/backend/secret_finder.py"
    click DBHandler "https://github.com/super50nic/project_elf_owl/blob/main/elfowl/backend/database_handler.py"

    click Dashboard "https://github.com/super50nic/project_elf_owl/tree/main/elfowl/dashboard/"
    click VulnTotal "https://github.com/super50nic/project_elf_owl/blob/main/vulntotal/vulntotal_cli.py"

    %% Styles
    classDef external fill:#dddddd,stroke:#999;
    classDef ingestion fill:#ccffcc,stroke:#393;
    classDef web fill:#ffcc99,stroke:#c60;
    classDef engine fill:#cc99ff,stroke:#63c;
    classDef db fill:#99ccff,stroke:#339;
    classDef cli fill:#e0ccff,stroke:#63c;
    classDef infra fill:#cccccc,stroke:#666;
```
