How matthew-ngzc/travel-info-extractor works
Project Type and Purpose
• This is a modular, containerized media-processing pipeline composed of discrete microservices.
• Each service (media-processor, transcription, ocr) performs one responsibility:
• media-processor: downloads or ingests raw media (audio/video) and extracts segments or metadata
• transcription: runs Whisper-based transcription on extracted audio
• ocr: (based on README) performs optical character recognition on image or video frames
• A shared “data” directory holds job artifacts and intermediate files.
File-Structure Analysis
Top level:
• README.md – describes folder layout
• .gitignore
Sub-folders (each is a standalone containerized service):
• media-processor/
– Dockerfile, docker-compose.yml, main.py, pyproject.toml, etc.
• transcription/
– Dockerfile, docker-compose.yml, app/ (main.py, models.py, transcription.py), pyproject.toml
• (ocr/) – similar pattern, not shown in tree
• data/ (git-ignored) – shared volume for artifacts
Patterns:
• Each service is in Python, uses Poetry or pyproject, runs in its own container.
• Docker Compose files in each service; likely orchestrated together at root via another docker-compose or script.
Key Architectural Insights from README
• A pipeline of services, loosely coupled and communicating via a shared filesystem (“data” folder) or potentially message queues.
• Each step picks up “jobs” from the shared directory, processes, and writes outputs back.
a. Main Components
• Ingress (job submission client or scheduler)
• media-processor service (container)
• transcription service (container)
• ocr service (container)
• Shared Data Volume (persistent storage)
• Optional Orchestrator (root-level docker-compose or workflow engine)
• External dependencies (e.g., Whisper model artifacts, external media sources)
b. Relationships & Data Flow
1. Ingress submits media download request → media-processor
2. media-processor pulls media, extracts audio/video segments → writes to Shared Data Volume
3. transcription watches Volume → picks up audio segments → runs Whisper → writes transcript JSON to Volume
4. ocr watches Volume → picks up image frames → runs OCR → writes text/artifacts to Volume
5. Downstream consumer (reporter or UI) reads final artifacts
c. Architectural Patterns & Principles
• Microservices: each service is isolated, single responsibility
• Containerization: each service in its own Docker container, with service-specific Dockerfile + Compose
• Shared-volume integration or message-driven pipeline (filesystem as queue)
• Clear separation of concerns: ingestion, processing, transcription, OCR
d. Technologies & Frameworks
• Python 3.x, Poetry/pyproject.toml dependency management
• Docker, Docker Compose for container orchestration
• Whisper (OpenAI) for transcription
• OCR library (e.g., Tesseract) for ocr service
• FastAPI or simple Python scripts (main.py) as service entrypoints
Final Advice
• Every component—even scripts like main.py can be shown as the service entrypoint.
• Don’t over–generalize: show both Dockerfiles and docker-compose.yml as part of each service’s boundary.
• Make data flows explicit, showing how jobs move through media-processor → transcription → ocr.