How twais/credable_interview works
Project Type and Purpose
• The project is a Python-based assessment that involves ingesting, cleaning, and serving CVE (Common Vulnerability Exposure) data.
• It includes data ingestion (via SFTP), a pipeline for processing the CSV file (cleaning duplicates, blank lines, etc.), and an API service built on FastAPI to expose the processed data.
• The README also indicates integration with an external trial SFTP server and usage of environment variables for configuration, along with testing support via Pytest.
Break Down the Main Components
• Data Ingestion Component:
– Represented by the tasks/sftp_ingest.py module.
– This module is responsible for connecting to the SFTP server (using credentials exported as environment variables) and retrieving the raw CSV data.
• Data Processing Pipeline:
– Split into parts like the tasks/csv_cleanup.py module which handles the cleaning of CSV files.
– The tasks/pipeline.py file orchestrates the overall process: reading the CSV input, cleaning up the duplicate and blank entries, and preparing the data.
• API Service:
– Defined in the tasks/api.py module and launched using uvicorn.
– It provides endpoints for querying the processed CVE data (e.g., by discovery date, with limits and cursors).
• Testing Suite:
– A set of tests found in the tests directory (tests/api.py and tests/test_*.py) to ensure the reliability of the different modules.
Relationships and Interactions
• The Pipeline Process:
– Takes an input CSV file (or downloads it via SFTP) and sends it through the CSV cleanup processing.
– The cleaned data is then likely stored in-memory (or possibly a transient data structure) ready for API access.
• The API Component:
– When the API is launched, it exposes endpoints that serve the processed data.
– It acts as a consumer of the data produced by the pipeline.
• Data Flow:
– Data flows from ingestion (SFTP ingestion) → processing (CSV cleanup and pipeline orchestration) → serving (API endpoints).
• Testing:
– The tests validate components like API responses and CSV cleanup logic, ensuring that each segment properly handles its responsibilities.
Architectural Patterns and Technologies
• The design follows a modular architecture – each distinct functionality (ingestion, processing, API) is encapsulated in its own module.
• FastAPI (and Uvicorn) is used to expose RESTful endpoints.
• The use of environment variables for configuration aligns with the Twelve-Factor App methodology.
• Testing via Pytest suggests an emphasis on modularity and testability.
– At the left, “External Data Source (SFTP Server)” connected via the “SFTP Ingestion” module.
– Next, the “Pipeline” which incorporates the “CSV Cleanup” stage to process data.
– Then, a separate “API Service” component that serves the cleaned data to the user.
– Include a “Testing” section to indicate how each module is verified.
• Clarity, correct directional flows, and detailed labels to capture the essential architectural elements.