# anasaito/skillner architecture

> Architecture diagram of the GitHub repository anasaito/skillner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anasaito/skillner
- Repository: https://github.com/anasaito/skillner

## Overview

##### Main Components Identification

The system should be represented with these key components:

a) Core Processing Layer:
- SkillExtractor (main class)
- Text Processor
- Matcher System
- Cleaner Module

b) Data Layer:
- EMSI Skills Database Integration
- Token Distribution Storage
- Skills Database Cache

c) External Dependencies:
- SpaCy NLP Engine
- PhraseMatcher

##### Component Relationships

**Input Text -> Text Processor -> Matcher System -> Skill Extractor -> Annotated Output**
- ↑               ↑

- SpaCy NLP     Skills Database

##### Specific Implementation Details

a) Core Components:
- `text_class.Text` for text processing
- `matcher_class.SkillsGetter` for skill matching
- `cleaner.Cleaner` for text preprocessing

b) Data Flow:
- Input text flow through the system
- Database queries and caching
- Demonstrate the annotation process
- Visualization output generation

**[Input Layer]**
- ↓
**[Processing Layer]**
- ↓
**[Storage Layer]**
- ↓
**[Output Layer]**

##### Key Technical Elements

- SpaCy integration point
- PhraseMatcher implementation
- EMSI database connection
- Skill annotation pipeline
- Visualization components
- API endpoints (if applicable)

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        IT[Input Text]
    end

    subgraph "Core Processing Layer"
        SE["SkillExtractor"]:::core
        TP["Text Processor"]:::core
        MS["Matcher System"]:::core
        CM["Cleaner Module"]:::core
    end

    subgraph "Data Layer"
        DB[("EMSI Skills DB")]:::storage
        TD[("Token Distribution")]:::storage
        SC[("Skills Cache")]:::storage
    end

    subgraph "External Dependencies"
        SpaCy["SpaCy NLP Engine"]:::external
        PM["PhraseMatcher"]:::external
    end

    subgraph "Processing Tools"
        RDF["Raw Data Fetcher"]:::tools
        TDC["Token Distribution Creator"]:::tools
        SDC["Surface Database Creator"]:::tools
    end

    subgraph "Visualization"
        VC["Visualization Components"]:::viz
    end

    %% Core Flow
    IT --> CM
    CM --> TP
    TP --> MS
    MS --> SE
    SE --> VC

    %% Dependencies
    SpaCy --> TP
    PM --> MS

    %% Data Connections
    DB --> MS
    TD --> MS
    SC --> SE

    %% Tool Connections
    RDF --> DB
    TDC --> TD
    SDC --> SC

    %% Styling
    classDef core fill:#2374ab
    classDef storage fill:#57a773
    classDef external fill:#ffd166
    classDef tools fill:#ef767a
    classDef viz fill:#7d3f98

    %% Click Events
    click SE "https://github.com/AnasAito/SkillNER/blob/master/skillNer/skill_extractor_class.py"
    click TP "https://github.com/AnasAito/SkillNER/blob/master/skillNer/text_class.py"
    click MS "https://github.com/AnasAito/SkillNER/blob/master/skillNer/matcher_class.py"
    click CM "https://github.com/AnasAito/SkillNER/blob/master/skillNer/cleaner.py"
    click DB "https://github.com/AnasAito/SkillNER/blob/master/skillNer/network/remote_db.py"
    click TD "https://github.com/AnasAito/SkillNER/blob/master/buckets/token_dist.json"
    click SC "https://github.com/AnasAito/SkillNER/blob/master/buckets/skills_processed.json"
    click VC "https://github.com/AnasAito/SkillNER/tree/master/skillNer/visualizer/"
    click RDF "https://github.com/AnasAito/SkillNER/blob/master/skills_processor/fetch_raw_data.py"
    click TDC "https://github.com/AnasAito/SkillNER/blob/master/skills_processor/create_token_dist.py"
    click SDC "https://github.com/AnasAito/SkillNER/blob/master/skills_processor/create_surf_db.py"
```
