How sohv/compliance-graph works
Based on a careful review of both the file tree and the README, this project is clearly a prototype for a financial compliance system that leverages knowledge graphs (with Neo4j) to extract, ingest, and validate compliance rules based on financial documents. The core idea is to extract relevant rules from documents, store them in a knowledge graph, and use semantic techniques (with SentenceTransformer NLP models) to assess transaction compliance.
Main Components
• Data Source & Document Input: Represent the source of your financial documents. Note the “data/docs” directory that holds financial documents (e.g., Synthetic_Financial_Rules.pdf).
• Rule Extraction: Highlight the process and modules responsible for extracting rules from PDFs. This involves the file “src/read_pdf.py” and “src/extract_rules.py” which likely use text processing and NLP techniques.
• Database Access & Ingestion: Show the process of taking the extracted rules and storing them in the Neo4j knowledge graph. The “src/db/connection.py” (and the entire “src/db” directory) is key here, as well as “src/ingest.py” which performs the ingestion.
• Compliance Module: Identify the functionality that retrieves rules from the knowledge graph and checks them against transactions. Use the “src/compliance/” directory components:
• rule_retrieve.py – retrieves relevant rules using semantic similarity.
• rule_check.py – evaluates if a transaction meets the compliance rules.
• check_compliance.py – orchestrates the overall compliance check.
• Evaluation and Search: Include evaluation (“src/evaluate_rules.py”) that assesses rule matching performance, and the “src/search_rules.py” which might be used for querying/assessing existing rules.
Relationships and Data Flow
• The extracted rules (saved as JSON files in the “data” directory) flow into the ingestion phase where they are loaded into the Neo4j knowledge graph (illustrate a connection from extraction module to database ingestion module).
• Once rules are in the graph, the compliance module retrieves these rules using semantic similarity (by leveraging the SentenceTransformer model) and checks incoming transactions for compliance.
• Finally, an evaluation module assesses the performance of the rule matching, with output possibly being logged in “data/output.txt” or evaluated via “src/evaluate_rules.py”.
Architectural Patterns & Design Principles
• Separation of Concerns: Components such as extraction, ingestion, compliance checking, and evaluation are clearly separated into different scripts and modules.
• Modular Design: Each functional area (compliance, database, extraction) is organized into its own subdirectory and Python file, allowing independent development and testing.
• Externalized Configuration: Use of a “config/config.py” for system parameters and environment variables (detailed in the README via the .env file) emphasizes centralized configuration management.
• Data Persistence through a Knowledge Graph: Emphasize the Neo4j database integration as a key technology which underpins the entire compliance rule retrieval and verification process.
Relevant Technologies & Tools
• Neo4j: As the target database for the knowledge graph.
• SentenceTransformer: For computing semantic similarity using the ‘all-MiniLM-L6-v2’ NLP model.
• Python: The primary programming language with modules split across directories.
• Various utility scripts (for ingest, extraction, compliance checking, etc.) that form the processing pipeline.
• Each component should have:
• A clearly labeled rectangle (e.g., "PDF Extraction (src/read_pdf.py & src/extract_rules.py)").