How amuizab/protein-3d-structure-pipeline-web-interface works
Project Type & Purpose
• The project is a hybrid application with two distinct parts:
• A data pipeline that processes protein gene data, maps gene names to UniProt IDs, generates validated URLs, downloads PDB files, manages metadata via an SQLite database, and finally uploads the files to Google Cloud Storage.
• The overall purpose is to collect and process protein structure data and let users explore the structures via a web interface deployed on Google Cloud Run.
Analyzing the File Structure
• The top-level project contains two main directories:
• data_pipelines: Contains submodules like "datagenes", "downloadpdb", and "map2uniprot". Each submodule is responsible for a step in the pipeline:
▸ datagenes: Likely handles gene name collection and preprocessing.
▸ map2uniprot: Manages the extraction of UniProt IDs from gene names.
▸ downloadpdb: Responsible for downloading PDB files based on provided URLs.
▸ Additionally, there are main scripts (main.py, uploadtoGCS.py) and a Jupyter notebook (projectProteins.ipynb) to orchestrate pipeline execution.
• web_server: Contains the web interface code including a Docker configuration (Dockerfile.txt), HTML file (index.html), and Python script (main.py) that sets up the server and likely integrates the Mol* viewer.
• Other files: requirements.txt lists dependencies important for both backend and frontend operations.
Key Architectural Components
• Data Pipeline Components:
• Gene Data Processing Module: Represented by the “datagenes” component.
• UniProt ID Mapping Module: Represented by the “map2uniprot” component.
• PDB Download Module: Represented by the “downloadpdb” component.
• Orchestration and Execution Flow: The main.py and the Jupyter Notebook, which coordinate the pipeline steps.
• Storage Operation: The SQLite database (for metadata) and uploadtoGCS.py which handles uploading to Google Cloud Storage.
• Web Server Components:
• Frontend Interface: HTML (index.html) for user interaction, including search and display elements.
• Backend Service: main.py in the web_server directory. This module manages API routes, integrates with the SQLite database for metadata, and loads PDB files (via links to Google Cloud Storage).
• Containerization: Dockerfile.txt and dockerignore.txt, which package the web server for deployment to Google Cloud Run.
• External Systems:
• Google Cloud Storage: Holds the PDB files.
• Google Cloud Run: Deployment platform for the web interface.
• SQLite Database: Local database holding metadata and relationships.
Elements
• Clear and descriptive labels for each component (e.g., “Gene Processing”, “UniProt Mapping”, “Downloader”, “Uploader to GCS”, “SQLite Metadata DB”, “Web API Server”, “Mol* Viewer UI”).
Final Advice
• Be as detailed as possible when separating the architecture into its core modules. The more granular you are, the clearer it will be for stakeholders to understand the multistage pipeline and its integration with the web service.
• Avoid oversimplifying; given the project’s complexity (over 17,300 PDB files processed, multiple integration points with Google Cloud, and real-time web interaction for 3D visualization), it is critical that every key component and dependency is noted.