# joon-solutions/felix-the-field-agent architecture

> Architecture diagram of the GitHub repository joon-solutions/felix-the-field-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joon-solutions/felix-the-field-agent
- Repository: https://github.com/joon-solutions/felix-the-field-agent

## Overview

##### Project type and purpose
- This is a Looker “Block” (pre-built module) combined with a Python-based ETL pipeline.
- Purpose: extract Looker System Activity and LookML metadata, load it into BigQuery, then provide a suite of LookML dashboards/explores for field-usage analytics and optimization insights.

##### File structure
- Top level:
- extraction/ – Python ETL pipeline pulling from Looker API and LookML repos
- dashboards/, explores/, views/ – LookML artifacts defining dashboards, explores, and views for the Block
- manifest.lkml, marketplace.json – Looker project metadata
- assets/ – static assets (images, docs)
- extraction/ breakdown:
- main.py – orchestration entrypoint
- requirements.txt, sample.env, sample.looker.ini – environment & dependency configs
- utils/ – helper modules for BigQuery, Looker API, repo parsing
- LookML breakdown:
- views/derived_tables/ – SQL-based derived tables loaded into BigQuery
- views/refined/ – modeling layers refining raw derived tables
- dashboards/ & explores/ – high-level user interfaces
- No frontend code – it is all Looker server-side.

##### README insights
- ETL pipeline writes to BigQuery permanent tables.
- Dashboards and Explores in Looker query these tables.
- Field usage metrics, dynamic field detection, best-practice violations.

a. Main components
1. Looker Platform (hosted Looker instance)
2. Extraction Service (Python app)
3. BigQuery Data Warehouse
4. LookML Project (this Block)
5. Version Control (Git/GitHub)
6. Users (Analysts/Developers)
b. Relationships & data flow
1. Extraction Service → (1) calls Looker API → fetches System Activity & LookML metadata
2. Extraction Service → ETL → loads/updates BigQuery tables
3. LookML Project → queries BigQuery tables via Looker SQL runner
4. Looker Platform → renders Dashboards & Explores defined in LookML
5. Users → interact with dashboards in Looker UI
6. Developers → push changes to LookML & extraction code via Git → deploy to Looker/GCP
c. Architectural patterns & principles
- ETL microservice: single responsibility for data extraction and load
- Data modeling layers: raw derived tables → refined views → dashboards/explores
- Infrastructure as code: LookML & Python code in Git, sample env for reproducibility
- Separation of concerns: extraction vs. analytics vs. presentation
d. Key technologies
- Python 3 + utils for Looker API & BigQuery (google-cloud-bigquery)
- BigQuery for data storage & derived tables
- LookML for modeling & dashboard definitions
- GitHub/Git for version control
- Looker API and Looker System Activity model

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Actors"
        U[("Users")]:::user
        D[("Developers")]:::dev
    end

    subgraph "Version Control"
        VC[("GitHub Repo")]:::code
    end

    subgraph "Extraction Service"
        ES[("Extraction Service")]:::service
        Main["main.py"]:::code
        Req["requirements.txt"]:::code
        Env["sample.env"]:::code
        Ini["sample.looker.ini"]:::code
        SPJ["sample.looker_project.json"]:::code
        SRT["sample.repos.txt"]:::code
        SCH["schema.yaml"]:::code
        subgraph "utils"
            BG["bigquery_worker.py"]:::code
            LW["looker_worker.py"]:::code
            LR["lookml_repo.py"]:::code
            HP["helpers.py"]:::code
            EN["enums.py"]:::code
            WK["worker.py"]:::code
        end
    end

    DB[("BigQuery")]:::db

    subgraph "Looker Platform"
        LP[("Looker Instance")]:::service
        subgraph "LookML Project"
            MAN["manifest.lkml"]:::code
            MOD["felix_the_field_agent.model.lkml"]:::code
            MP["marketplace.json"]:::code
            SH["install.sh"]:::code
            subgraph "Dashboards"
                D1["dynamic_field_analysis.dashboard.lookml"]:::code
                D2["fields_usage_in_queries.dashboard.lookml"]:::code
                D3["fields_without_best_practices_configurations.dashboard.lookml"]:::code
                D4["impact_analysis_of_field_usage.dashboard.lookml"]:::code
                D5["monitoring_unused_fields.dashboard.lookml"]:::code
            end
            subgraph "Explores"
                E1["field_usage.explore.lkml"]:::code
                E2["query_metrics.explore.lkml"]:::code
            end
            subgraph "Views"
                subgraph "Derived Tables"
                    DT1["field_usage.derived.view.lkml"]:::code
                    DT2["lookml_fields.derived.view.lkml"]:::code
                    DT3["parsed_query.derived.view.lkml"]:::code
                    DT4["query_dynamic_field.derived.view.lkml"]:::code
                    DT5["unused_fields.derived.view.lkml"]:::code
                    DT6["user_email.derived.view.lkml"]:::code
                end
                subgraph "Refined Views"
                    RV1["history.refined.view.lkml"]:::code
                    RV2["query_metrics.refined.view.lkml"]:::code
                end
                subgraph "Core Views"
                    CV1["dashboard.view.lkml"]:::code
                    CV2["explore_label.view.lkml"]:::code
                    CV3["group.view.lkml"]:::code
                    CV4["group_user.view.lkml"]:::code
                    CV5["history.view.lkml"]:::code
                    CV6["look.view.lkml"]:::code
                    CV7["query.view.lkml"]:::code
                    CV8["query_metrics.view.lkml"]:::code
                    CV9["role.view.lkml"]:::code
                    CV10["user.view.lkml"]:::code
                    CV11["user_facts.view.lkml"]:::code
                    CV12["user_facts_role.view.lkml"]:::code
                end
            end
            subgraph "Assets"
                AR["assets/README"]:::code
                AD["assets/"]:::code
            end
        end
    end

    U -->|"view dashboards"| LP
    D -->|"push code"| VC
    VC -->|"deploy extraction code"| ES
    VC -->|"deploy LookML"| LP
    ES -->|"calls Looker API"| LP
    ES -->|"loads data"| DB
    MAN -->|"queries"| DB
    MOD -->|"queries"| DB
    LP -->|"renders dashboards"| U

    click ES "https://github.com/joon-solutions/felix-the-field-agent/tree/main/extraction/"
    click Main "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/main.py"
    click Req "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/requirements.txt"
    click Env "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/sample.env"
    click Ini "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/sample.looker.ini"
    click SPJ "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/sample.looker_project.json"
    click SRT "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/sample.repos.txt"
    click SCH "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/schema.yaml"
    click BG "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/utils/bigquery_worker.py"
    click LW "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/utils/looker_worker.py"
    click LR "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/utils/lookml_repo.py"
    click HP "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/utils/helpers.py"
    click EN "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/utils/enums.py"
    click WK "https://github.com/joon-solutions/felix-the-field-agent/blob/main/extraction/utils/worker.py"
    click MAN "https://github.com/joon-solutions/felix-the-field-agent/blob/main/manifest.lkml"
    click MOD "https://github.com/joon-solutions/felix-the-field-agent/blob/main/felix_the_field_agent.model.lkml"
    click MP "https://github.com/joon-solutions/felix-the-field-agent/blob/main/marketplace.json"
    click SH "https://github.com/joon-solutions/felix-the-field-agent/blob/main/install.sh"
    click D1 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/dashboards/dynamic_field_analysis.dashboard.lookml"
    click D2 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/dashboards/fields_usage_in_queries.dashboard.lookml"
    click D3 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/dashboards/fields_without_best_practices_configurations.dashboard.lookml"
    click D4 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/dashboards/impact_analysis_of_field_usage.dashboard.lookml"
    click D5 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/dashboards/monitoring_unused_fields.dashboard.lookml"
    click E1 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/explores/field_usage.explore.lkml"
    click E2 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/explores/query_metrics.explore.lkml"
    click DT1 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/derived_tables/field_usage.derived.view.lkml"
    click DT2 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/derived_tables/lookml_fields.derived.view.lkml"
    click DT3 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/derived_tables/parsed_query.derived.view.lkml"
    click DT4 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/derived_tables/query_dynamic_field.derived.view.lkml"
    click DT5 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/derived_tables/unused_fields.derived.view.lkml"
    click DT6 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/derived_tables/user_email.derived.view.lkml"
    click RV1 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/refined/history.refined.view.lkml"
    click RV2 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/refined/query_metrics.refined.view.lkml"
    click CV1 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/dashboard.view.lkml"
    click CV2 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/explore_label.view.lkml"
    click CV3 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/group.view.lkml"
    click CV4 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/group_user.view.lkml"
    click CV5 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/history.view.lkml"
    click CV6 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/look.view.lkml"
    click CV7 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/query.view.lkml"
    click CV8 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/query_metrics.view.lkml"
    click CV9 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/role.view.lkml"
    click CV10 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/user.view.lkml"
    click CV11 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/user_facts.view.lkml"
    click CV12 "https://github.com/joon-solutions/felix-the-field-agent/blob/main/views/user_facts_role.view.lkml"
    click AR "https://github.com/joon-solutions/felix-the-field-agent/tree/main/assets/README"
    click AD "https://github.com/joon-solutions/felix-the-field-agent/tree/main/assets/"

    classDef service fill:#f9f,stroke:#333,stroke-width:1px
    classDef db fill:#ccf,stroke:#333,stroke-width:1px
    classDef code fill:#efe,stroke:#333,stroke-width:1px
    classDef user fill:#fcf,stroke:#333,stroke-width:1px
    classDef dev fill:#cff,stroke:#333,stroke-width:1px
```
