# hshashank06/gaidp-blizzard architecture

> Architecture diagram of the GitHub repository hshashank06/gaidp-blizzard, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hshashank06/gaidp-blizzard
- Repository: https://github.com/hshashank06/gaidp-blizzard

## Overview

##### Project type and purpose
- This is an AI-driven data-profiling and regulatory compliance pipeline, implemented as a collection of Jupyter notebooks and scripts.
- It extracts structured data from PDF reports, builds semantic search indexes, dynamically generates compliance rules, and assigns risk scores via anomaly detection and LLM-based reasoning. A Gradio UI front-end provides real-time interaction.

##### File structure
- code/data/synthetic-data-generation: pipeline to produce training/test data (GPT-4 prompts, embedding similarity).
- code/models: three submodules
- cross-encoder (reranking)
- embedder-fine-tuning (custom sentence-transformer)
- unsupervised-learning (Isolation Forest)
- code/src: core pipelines
- automated-rules-generation (LangChain + LangGraph workflows)
- risk-analysis (risk scoring notebooks)
- code/test: sample regulatory PDFs
- artifacts/arch: design docs (initial + final)
- README.md: end-to-end workflow and tech stack

##### README for additional insights
- Five stages: PDF parsing, vector embeddings & retrieval (Pinecone/FAISS), reranking with cross-encoder, dynamic rule generation (KeyBERT + LangChain/Graph), customer risk profiling (Isolation Forest + LLM scoring), plus external data (SerpAPI) and synthetic data.
- Gradio UI sits atop for user queries.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data & Storage Layer
    subgraph "Data & Storage" 
        direction TB
        PDFs["Regulatory PDFs"]:::data
        SerpAPI["External Data (SerpAPI)"]:::ext
        VectorStore["Vector Store\n(Pinecone / FAISS)"]:::data
        Artifacts["Design Docs"]:::data
    end

    %% Data Preparation Layer
    subgraph "Data Preparation"
        direction TB
        Parsing["PDF Parsing\n(PyMuPDF, pdfplumber)"]:::data
        Metadata["Metadata Filtering"]:::data
    end

    %% Model Training Layer
    subgraph "Model Training"
        direction TB
        SynData["Synthetic Data Generator"]:::ml
        EmbedTrain["Embedding Fine-Tuning\n(Sentence-Transformers)"]:::ml
        CrossTrain["Cross-Encoder Training"]:::ml
        IsoLearn["Isolation Forest Training"]:::ml
    end

    %% Orchestration Layer
    subgraph "Orchestration\n(LangChain & LangGraph)" 
        direction TB
        Workflow["Agentic Workflows"]:::orch
    end

    %% Runtime Inference Layer
    subgraph "Runtime Inference"
        direction TB
        EmbedGen["Embedding Generation"]:::ml
        Retrieval["Vector Retrieval"]:::ml
        Rerank["Reranking\n(Cross-Encoder)"]:::ml
        RuleGen["Rule Generation\n(KeyBERT + LangChain)"]:::orch
        RiskScore["Risk Analysis\n(Isolation Forest + LLM)"]:::ml
    end

    %% UI Layer
    subgraph "UI & External"
        direction TB
        Gradio["Gradio UI"]:::ext
    end

    %% Flow Connections
    PDFs --> Parsing
    Parsing --> Metadata
    Metadata --> EmbedGen
    SerpAPI --> RuleGen
    Metadata -->|index| VectorStore
    EmbedGen --> Retrieval
    Retrieval --> Rerank
    Rerank --> RuleGen
    SynData --> EmbedTrain
    SynData --> IsoLearn
    EmbedTrain --> EmbedGen
    IsoLearn --> RiskScore
    RuleGen --> RiskScore
    Workflow --> RuleGen
    Workflow --> RiskScore
    RiskScore --> Gradio

    %% Alternative vector store depiction
    VectorStore -. alternative .-> Retrieval

    %% Artifacts reference
    Artifacts -. refers to .-> Parsing

    %% Click Events
    click PDFs "https://github.com/hshashank06/gaidp-blizzard/tree/main/code/test/"
    click SynData "https://github.com/hshashank06/gaidp-blizzard/blob/main/code/data/synthetic-data-generation/Synthetic_Data_Generator.ipynb"
    click EmbedTrain "https://github.com/hshashank06/gaidp-blizzard/blob/main/code/models/embedder-fine-tuning/Fine_Tuning.ipynb"
    click CrossTrain "https://github.com/hshashank06/gaidp-blizzard/blob/main/code/models/cross-encoder/Cross_Encoder.ipynb"
    click RuleGen "https://github.com/hshashank06/gaidp-blizzard/blob/main/code/src/automated-rules-generation/Automated_Rules_Generation.ipynb"
    click RiskScore "https://github.com/hshashank06/gaidp-blizzard/blob/main/code/src/risk-analysis/Risk_Analysis.ipynb"
    click IsoLearn "https://github.com/hshashank06/gaidp-blizzard/blob/main/code/models/unsupervised-learning/isolation_forest_model.joblib"
    click Gradio "https://github.com/hshashank06/gaidp-blizzard/blob/main/README.md"
    click Artifacts "https://github.com/hshashank06/gaidp-blizzard/tree/main/artifacts/arch/"

    %% Styling
    classDef data fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef ml fill:#c8e6c9,stroke:#43a047,color:#1b5e20
    classDef orch fill:#ffe0b2,stroke:#fb8c00,color:#e65100
    classDef ext fill:#e1bee7,stroke:#8e24aa,color:#4a148c
```
