# google/facade architecture

> Architecture diagram of the GitHub repository google/facade, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google/facade
- Repository: https://github.com/google/facade

## Overview

##### Project Type and Purpose
- This is a reference implementation of FACADE, a deep-learning-based contextual anomaly detection system in Python.
- It processes two raw event sources (actions and contexts), featurizes them, merges features, then trains and runs inference on a neural model to produce anomaly scores.

##### File Structure
- Top-level “action” and “context” packages: extract raw events from logs.
- “batch” package: three entrypoints—dataset_maker_main (build training/validation TFRecords), inference_main (run scoring), read_scores_main (format output).
- “pipelines” package: three pipeline modules—context_featurizer, history_featurizer, merger—each implements one stage of feature engineering.
- “model” package: input pipeline (ingest TFRecords), architectures (network definitions), layers (custom TF layers), loss, metrics, optimization, and train_main.
- “protos” directory: .proto definitions and generated Python classes for messages (action, context, directive, config, score, vocab, etc.).
- “common” package: shared utilities (directive parsing, time utilities, vocab builder, TF Example conversion).
- Config files (.textproto under model/input, pipeline tests) drive parametrization.

##### README for Insights
- Data is stored and exchanged as TFRecord.
- A three-step batch workflow: generate TFRecords, train/evaluate model, run inference and read top scores.
- All heavy logic is isolated in reusable modules, making it easy to plug in alternate featurizers or architectures.

##### Main Components and Their Relationships
A. Data Sources
- Action Source ← sample/action.tfrecord
- Context Source ← sample/context.tfrecord

B. Protobuf Definitions
- action.proto, context.proto, directive.proto, config.proto, etc.
- Used by “action”, “context”, “common” and across pipeline and model packages.

C. Batch Processing Layer
- dataset_maker_main
– Uses action_source and context_source to read raw events
– Invokes pipelines/context_featurizer → history_featurizer → merger
– Emits train/validation TFRecords
- inference_main
– Reads raw events, applies same pipelines, loads trained model, outputs score TFRecords
- read_scores_main
– Reads score TFRecords, outputs human-readable top anomalies

D. Feature Engineering Pipelines
- context_featurizer: extracts contextual features from code-review logs
- history_featurizer: computes user history summary features (e.g., heaviest items)
- merger: joins features into a single tf.Example per event pair

E. Model Layer
- Input Pipeline (model/input/pipeline.py) reads TFRecords into tf.Dataset
- Architectures (concatenate_then_snn, token_embeddings): build computation graphs
- Layers (snn, oov_dropout, segment_embedder, etc.), Loss (pairwise_huber, multi_similarity, etc.), Metrics, Optimizer
- train_main: trains or evaluates model, writes checkpoints to model_dir

F. Outputs
- Model Artifacts: saved checkpoints, SavedModel or TF checkpoint files
- Scores: TFRecord of anomaly scores + metadata

##### Architectural Patterns and Design Principles
- Modular separation of concerns: raw-source → pipeline → model → inference
- Pipeline pattern: each pipeline stage is testable and independently pluggable
- Protobuf-driven configuration and data interchange ensures type safety and language neutrality
- Use of TFRecord + tf.data for scalable I/O
- Test-first approach: nearly every module has a corresponding _test.py

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        ActionSource["Action Source"]:::process
        ContextSource["Context Source"]:::process
    end

    %% Protobuf & Config Layer
    subgraph "Configuration & Protobuf" 
        direction TB
        Protos["Protobuf Definitions"]:::config
        ModelConfig["Model Config Textproto"]:::config
        InputConfig["Input Pipeline Test Config"]:::config
    end

    %% Utilities
    subgraph "Utilities"
        direction TB
        Common["Common Utilities"]:::process
    end

    %% Batch Processing Layer
    subgraph "Batch Processing"
        direction TB
        DatasetMaker["dataset_maker_main"]:::process
        Inference["inference_main"]:::process
        ReadScores["read_scores_main"]:::process
        BatchLib["Batch Helper Library"]:::process
    end

    %% Pipeline Stages
    subgraph "Feature Engineering Pipelines"
        direction TB
        ContextFeat["context_featurizer"]:::process
        HistoryFeat["history_featurizer"]:::process
        Merger["merger"]:::process
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        InputPipe["Input Pipeline"]:::process
        Arch1["concatenate_then_snn"]:::process
        Arch2["token_embeddings"]:::process
        Layers["Custom Layers"]:::process
        Loss["Loss Functions"]:::process
        Metrics["Metrics"]:::process
        Opt["Optimization"]:::process
        TrainMain["train_main"]:::process
        ModelCore["Core Model Definition"]:::process
    end

    %% Data Stores & Outputs
    subgraph "Data Stores & Outputs"
        direction TB
        TFTrain["Train/Validation TFRecords"]:::dataStore
        TFScore["Score TFRecords"]:::dataStore
        ModelDir["Model Directory (Checkpoints/SavedModel)"]:::dataStore
        AnomalyScores["Human-readable Anomaly Scores"]:::dataStore
    end

    %% Data Flows
    ActionSource -->|raw events| DatasetMaker
    ContextSource -->|raw events| DatasetMaker
    DatasetMaker -->|tf.Example| ContextFeat
    ContextFeat -->|tf.Example| HistoryFeat
    HistoryFeat -->|tf.Example| Merger
    Merger -->|TFRecord| TFTrain
    TFTrain -->|tf.data| InputPipe
    InputPipe -->|Dataset| TrainMain
    TrainMain -->|checkpoints| ModelDir

    %% Inference Flow
    ActionSource -->|raw events| Inference
    ContextSource -->|raw events| Inference
    Inference -->|tf.Example| ContextFeat
    Inference -->|tf.Example| HistoryFeat
    Inference -->|tf.Example| Merger
    Merger -->|Score TFRecord| TFScore
    TFScore -->|records| ReadScores
    ReadScores -->|top anomalies| AnomalyScores

    %% Dependencies
    Protos -.-> DatasetMaker
    Protos -.-> Inference
    Protos -.-> ContextFeat
    Protos -.-> HistoryFeat
    Protos -.-> Merger
    Protos -.-> InputPipe
    Common -.-> DatasetMaker
    Common -.-> Inference
    Common -.-> ContextFeat
    Common -.-> HistoryFeat
    Common -.-> Merger
    Common -.-> TrainMain
    ModelConfig -.-> TrainMain
    InputConfig -.-> InputPipe

    %% Click Events
    click ActionSource "https://github.com/google/facade/blob/main/action/action_source.py"
    click ContextSource "https://github.com/google/facade/blob/main/context/context_source.py"
    click Protos "https://github.com/google/facade/tree/main/protos/"
    click Common "https://github.com/google/facade/tree/main/common/"
    click DatasetMaker "https://github.com/google/facade/blob/main/batch/dataset_maker_main.py"
    click Inference "https://github.com/google/facade/blob/main/batch/inference_main.py"
    click ReadScores "https://github.com/google/facade/blob/main/batch/read_scores_main.py"
    click BatchLib "https://github.com/google/facade/blob/main/batch/batch_lib.py"
    click ContextFeat "https://github.com/google/facade/blob/main/pipelines/context_featurizer/pipeline.py"
    click HistoryFeat "https://github.com/google/facade/blob/main/pipelines/history_featurizer/pipeline.py"
    click Merger "https://github.com/google/facade/blob/main/pipelines/merger/pipeline.py"
    click InputPipe "https://github.com/google/facade/blob/main/model/input/pipeline.py"
    click Arch1 "https://github.com/google/facade/blob/main/model/architectures/concatenate_then_snn.py"
    click Arch2 "https://github.com/google/facade/blob/main/model/architectures/token_embeddings.py"
    click Layers "https://github.com/google/facade/tree/main/model/layers/"
    click Loss "https://github.com/google/facade/tree/main/model/loss/"
    click Metrics "https://github.com/google/facade/tree/main/model/metrics/"
    click Opt "https://github.com/google/facade/tree/main/model/optimization/"
    click TrainMain "https://github.com/google/facade/blob/main/model/train_main.py"
    click ModelCore "https://github.com/google/facade/blob/main/model/model.py"
    click ModelConfig "https://github.com/google/facade/blob/main/model/test_model_config.textproto"
    click InputConfig "https://github.com/google/facade/blob/main/model/input/test_config.textproto"

    %% Styles
    classDef dataStore fill:#bbdefb,stroke:#0d47a1,color:#0d47a1
    classDef process fill:#c8e6c9,stroke:#1b5e20,color:#1b5e20
    classDef config fill:#fff9c4,stroke:#f57f17,color:#f57f17
```
