How joon-solutions/felix-the-field-agent works
Project type and purpose
• This is a Looker “Block” (pre-built module) combined with a Python-based ETL pipeline.
• Purpose: extract Looker System Activity and LookML metadata, load it into BigQuery, then provide a suite of LookML dashboards/explores for field-usage analytics and optimization insights.
File structure
• Top level:
• extraction/ – Python ETL pipeline pulling from Looker API and LookML repos
• dashboards/, explores/, views/ – LookML artifacts defining dashboards, explores, and views for the Block
• manifest.lkml, marketplace.json – Looker project metadata
• assets/ – static assets (images, docs)
• extraction/ breakdown:
• main.py – orchestration entrypoint
• requirements.txt, sample.env, sample.looker.ini – environment & dependency configs
• utils/ – helper modules for BigQuery, Looker API, repo parsing
• LookML breakdown:
• views/derived_tables/ – SQL-based derived tables loaded into BigQuery
• views/refined/ – modeling layers refining raw derived tables
• dashboards/ & explores/ – high-level user interfaces
• No frontend code – it is all Looker server-side.
README insights
• ETL pipeline writes to BigQuery permanent tables.
• Dashboards and Explores in Looker query these tables.
• Field usage metrics, dynamic field detection, best-practice violations.
a. Main components
1. Looker Platform (hosted Looker instance)
2. Extraction Service (Python app)
3. BigQuery Data Warehouse
4. LookML Project (this Block)
5. Version Control (Git/GitHub)
6. Users (Analysts/Developers)
b. Relationships & data flow
1. Extraction Service → (1) calls Looker API → fetches System Activity & LookML metadata
2. Extraction Service → ETL → loads/updates BigQuery tables
3. LookML Project → queries BigQuery tables via Looker SQL runner
4. Looker Platform → renders Dashboards & Explores defined in LookML
5. Users → interact with dashboards in Looker UI
6. Developers → push changes to LookML & extraction code via Git → deploy to Looker/GCP
c. Architectural patterns & principles
• ETL microservice: single responsibility for data extraction and load
• Data modeling layers: raw derived tables → refined views → dashboards/explores
• Infrastructure as code: LookML & Python code in Git, sample env for reproducibility
• Separation of concerns: extraction vs. analytics vs. presentation
d. Key technologies
• Python 3 + utils for Looker API & BigQuery (google-cloud-bigquery)
• BigQuery for data storage & derived tables
• LookML for modeling & dashboard definitions
• GitHub/Git for version control
• Looker API and Looker System Activity model