# varun-l/confl-loader architecture

> Architecture diagram of the GitHub repository varun-l/confl-loader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/varun-l/confl-loader
- Repository: https://github.com/varun-l/confl-loader

## Overview

##### Project type and purpose
- From the file tree (main.py, sk.py, sk_advanced.py) and README, this is a small Python-based integration tool (an “open-source CLI utility”) that:
- Loads credentials from a .env file
- Uses the Atlassian Python API to fetch documents from a Confluence (or similar) instance
- Parses content (via lxml)
- Feeds documents into LangChain pipelines for processing (summaries, embeddings, etc.)
- (Planned) Stores a “last fetched” timestamp and only pulls updated documents after that time

##### File structure
- .env – holds API_TOKEN and any other secrets
- requirements.txt – lists langchain-community, atlassian-python-api, python-dotenv, lxml
- main.py – the CLI entrypoint / orchestrator
- sk.py – core/simple implementation of the document-fetch + process logic
- sk_advanced.py – advanced/extended pipeline (e.g., additional LangChain chains or storage features)

##### README for additional insights
- The README shows essential pip installs and hints at future work: tracking “when” documents were last pulled and only retrieving updated docs.

##### Important details to capture
- Secrets/config reading step (.env → python-dotenv)
- Distinction between fetching raw document and parsing it
- The role of LangChain modules in sk.py vs. sk_advanced.py

- Highlights current vs. future functionality
- Indicates which external service (Atlassian) and which libraries power each step

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Nodes
    Entrypoint["CLI Entrypoint\n(main.py)"]:::infra
    Config["Config Loader\n(.env)"]:::infra
    APIClient["Atlassian API Client\n(sk.py)"]:::external
    AtlassianConfluence["Confluence API"]:::external
    Fetcher["Document Fetcher\n& Parser\n(sk.py)"]:::processing
    subgraph "Processing Pipelines"
        direction TB
        LangSimple["LangChain Simple Pipeline\n(sk.py)"]:::processing
        LangAdvanced["LangChain Advanced Pipeline\n(sk_advanced.py)"]:::processing
    end
    Storage["Output / Storage"]:::infra
    FutureTS["Timestamp Manager\n(future)"]:::infraFuture
    Requirements["requirements.txt"]:::infra
    README["README.md"]:::infra

    %% Data Flow
    Entrypoint -->|"loads config"| Config
    Config -->|"provides API_TOKEN"| APIClient
    FutureTS -.->|filters pulls| APIClient
    APIClient -->|"requests docs"| AtlassianConfluence
    AtlassianConfluence -->|"returns raw HTML/XML"| APIClient
    APIClient -->|"raw docs"| Fetcher
    Fetcher -->|"clean text"| LangSimple
    Fetcher -->|"clean text"| LangAdvanced
    LangSimple -->|"results"| Storage
    LangAdvanced -->|"results"| Storage

    %% Documentation
    Entrypoint --> Requirements
    Entrypoint --> README

    %% Click Events
    click Entrypoint "https://github.com/varun-l/confl-loader/blob/main/main.py"
    click Config "https://github.com/varun-l/confl-loader/blob/main/.env"
    click APIClient "https://github.com/varun-l/confl-loader/blob/main/sk.py"
    click Fetcher "https://github.com/varun-l/confl-loader/blob/main/sk.py"
    click LangSimple "https://github.com/varun-l/confl-loader/blob/main/sk.py"
    click LangAdvanced "https://github.com/varun-l/confl-loader/blob/main/sk_advanced.py"
    click Requirements "https://github.com/varun-l/confl-loader/blob/main/requirements.txt"
    click README "https://github.com/varun-l/confl-loader/blob/main/README.md"

    %% Styles
    classDef infra fill:#cfe2f3,stroke:#000,stroke-width:1px;
    classDef external fill:#d5ead1,stroke:#000,stroke-width:1px;
    classDef processing fill:#fce5cd,stroke:#000,stroke-width:1px;
    classDef infraFuture fill:#cfe2f3,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5;
```
