# pankajackson/fs2elastic architecture

> Architecture diagram of the GitHub repository pankajackson/fs2elastic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pankajackson/fs2elastic
- Repository: https://github.com/pankajackson/fs2elastic

## Overview

##### Project Overview
- FS2Elastic is a Python-based CLI tool designed to sync file system data (primarily CSV and similar file formats) to an Elasticsearch instance.
- Its key features include real‑time syncing and the use of a custom configuration file to set up parameters such as file paths, dataset details, and Elasticsearch connection settings.

##### Key Components
a. CLI Application (Entry Point):
– The user invokes the fs2elastic command (with or without a custom config), which starts the program.

b. Configuration Loader/Manager:
– This module (e.g., confbuilder.py) is responsible for loading and parsing the configuration file (fs2elastic.conf). The configuration governs app behavior, dataset locations, logging, and Elasticsearch connection properties.

c. Dataset Processor:
– The dataset_processor.py deals with scanning the dataset source directory, reading files (e.g., CSV, XLSX, JSON), possibly chunking the data, and preparing it for index insertion. It might also handle multi-threading/multi-worker operations, as hinted by configuration parameters (max_workers, threads_per_worker, chunk_size).

d. Elasticsearch Handler:
– The es_handler.py module manages the connectivity with Elasticsearch. It handles credentials, host information, index prefixes, and data transmission ensuring the data is indexed properly on the ES cluster.

e. Core Controller:
– The central module (fs2elastic.py) orchestrates the overall workflow. It ties together the loading of configurations, invocation of the dataset processing, and leveraging the ES handler for the data syncing process.

f. Logging Mechanism:
– Although the code for logging might be implemented within these modules (or via standard libraries), it is guided by LogConfig settings from the configuration file ensuring that logs are written to the specified file with rotation parameters.

##### External Entities
– Filesystem: Represents the source of the data (directories containing CSV, JSON, etc.).
– Elasticsearch Service: The external Elasticsearch cluster that receives the synced data.

##### Relationships and Data Flow
- The configuration module reads settings that define parameters for the dataset source directory, file type support, and worker/thread configurations.
- The Dataset Processor retrieves file-based data from the filesystem, processes and chunks the dataset, and then passes the processed data to the ES Handler.
- The ES Handler, using the configuration settings, establishes a connection to the Elasticsearch server and pumps the data into the specified index (which may use a prefix from ESConfig).
- Logging and error handling should be represented alongside these operations.

##### Architectural Patterns to Highlight
- Modularity: The separation of concerns is clear – configuration management, file processing, and Elasticsearch handling are encapsulated in separate modules.
- CLI-driven execution: Emphasize the command-line invocation as the primary entry point.
- Configurable data processing: Illustrate the configurability via the external configuration file that drives operational parameters.
- Real-time, streaming-like syncing pattern: The system continuously monitors or processes files from a source directory and streams them into Elasticsearch.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Internal FS2Elastic Components
    subgraph "FS2Elastic System"
        CLI["CLI Application"]:::internal
        Config["Configuration Loader"]:::internal
        Processor["Dataset Processor"]:::internal
        ESHandler["Elasticsearch Handler"]:::internal
        Logger["Logging Mechanism"]:::internal
    end

    %% External Systems
    subgraph "External Systems"
        FS["Filesystem"]:::external
        ES["Elasticsearch Server"]:::external
    end

    %% Data and Control Flow
    CLI -->|"LoadConfiguration"| Config
    Config -->|"ApplySettings"| Processor
    Config -->|"ApplySettings"| ESHandler
    FS -->|"DataInput(CSV,etc.)"| Processor
    Processor -->|"ProcessedData"| ESHandler
    ESHandler -->|"DataSync/Indexing"| ES

    %% Logging (shown as dashed flows)
    CLI -.->|"Logs"| Logger
    Config -.->|"Logs"| Logger
    Processor -.->|"Logs"| Logger
    ESHandler -.->|"Logs"| Logger

    %% Click Events for Internal Components
    click CLI "https://github.com/pankajackson/fs2elastic/blob/main/src/fs2elastic/fs2elastic.py"
    click Config "https://github.com/pankajackson/fs2elastic/blob/main/src/fs2elastic/confbuilder.py"
    click Processor "https://github.com/pankajackson/fs2elastic/blob/main/src/fs2elastic/dataset_processor.py"
    click ESHandler "https://github.com/pankajackson/fs2elastic/blob/main/src/fs2elastic/es_handler.py"

    %% Styling
    classDef internal fill:#d1e8ff,stroke:#0d47a1,stroke-width:2px;
    classDef external fill:#ffe0b2,stroke:#bf360c,stroke-width:2px;
    
    %% Arrow Styles
    linkStyle 0,1,2,3,4,5 stroke:#1565c0,stroke-width:2px;
    linkStyle 6,7,8,9 stroke:#ff6f00,stroke-dasharray: 5, 5, 0;
```
