How google/facade works
Project Type and Purpose
• This is a reference implementation of FACADE, a deep-learning-based contextual anomaly detection system in Python.
• It processes two raw event sources (actions and contexts), featurizes them, merges features, then trains and runs inference on a neural model to produce anomaly scores.
File Structure
• Top-level “action” and “context” packages: extract raw events from logs.
• “batch” package: three entrypoints—dataset_maker_main (build training/validation TFRecords), inference_main (run scoring), read_scores_main (format output).
• “pipelines” package: three pipeline modules—context_featurizer, history_featurizer, merger—each implements one stage of feature engineering.
• “model” package: input pipeline (ingest TFRecords), architectures (network definitions), layers (custom TF layers), loss, metrics, optimization, and train_main.
• “protos” directory: .proto definitions and generated Python classes for messages (action, context, directive, config, score, vocab, etc.).
• “common” package: shared utilities (directive parsing, time utilities, vocab builder, TF Example conversion).
• Config files (.textproto under model/input, pipeline tests) drive parametrization.
README for Insights
• Data is stored and exchanged as TFRecord.
• A three-step batch workflow: generate TFRecords, train/evaluate model, run inference and read top scores.
• All heavy logic is isolated in reusable modules, making it easy to plug in alternate featurizers or architectures.
Main Components and Their Relationships
A. Data Sources
• Action Source ← sample/action.tfrecord
• Context Source ← sample/context.tfrecord
B. Protobuf Definitions
• action.proto, context.proto, directive.proto, config.proto, etc.
• Used by “action”, “context”, “common” and across pipeline and model packages.
C. Batch Processing Layer
• dataset_maker_main
– Uses action_source and context_source to read raw events
– Invokes pipelines/context_featurizer → history_featurizer → merger
– Emits train/validation TFRecords
• inference_main
– Reads raw events, applies same pipelines, loads trained model, outputs score TFRecords
• read_scores_main
– Reads score TFRecords, outputs human-readable top anomalies
D. Feature Engineering Pipelines
• context_featurizer: extracts contextual features from code-review logs
• history_featurizer: computes user history summary features (e.g., heaviest items)
• merger: joins features into a single tf.Example per event pair
E. Model Layer
• Input Pipeline (model/input/pipeline.py) reads TFRecords into tf.Dataset
• Architectures (concatenate_then_snn, token_embeddings): build computation graphs
• Layers (snn, oov_dropout, segment_embedder, etc.), Loss (pairwise_huber, multi_similarity, etc.), Metrics, Optimizer
• train_main: trains or evaluates model, writes checkpoints to model_dir
F. Outputs
• Model Artifacts: saved checkpoints, SavedModel or TF checkpoint files
• Scores: TFRecord of anomaly scores + metadata
Architectural Patterns and Design Principles
• Modular separation of concerns: raw-source → pipeline → model → inference
• Pipeline pattern: each pipeline stage is testable and independently pluggable
• Protobuf-driven configuration and data interchange ensures type safety and language neutrality
• Use of TFRecord + tf.data for scalable I/O
• Test-first approach: nearly every module has a corresponding _test.py