# ternaustralia/vm_data_acquisition architecture

> Architecture diagram of the GitHub repository ternaustralia/vm_data_acquisition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ternaustralia/vm_data_acquisition
- Repository: https://github.com/ternaustralia/vm_data_acquisition

## Overview

VM_data_acquisition coordinates site-level logger-data acquisition, evaluation, and production of secondary data and metadata. Its principal workflow is task selection by site, followed by data processing or status evaluation and writing derived products; separate branches handle 10 Hz data, metadata, and RTMC configuration. The excerpts show several calls, while the source index supplies additional module references; links without direct call evidence are omitted. The README is brief, so some capability details and integration wiring remain uncertain.

Main entry points: [`tasks_manager.py`](https://github.com/ternaustralia/vm_data_acquisition/blob/master/tasks_manager.py) (Task manager), [`data_parser.py`](https://github.com/ternaustralia/vm_data_acquisition/blob/master/data_parser.py) (Site data parser), [`sparql_site_details.py`](https://github.com/ternaustralia/vm_data_acquisition/blob/master/sparql_site_details.py) (Site details), [`file_constructors.py`](https://github.com/ternaustralia/vm_data_acquisition/blob/master/file_constructors.py) (Data constructors)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Task orchestration"]
  node_tasks["Task manager<br/>[tasks_manager.py]"]
end

subgraph group_data["Logger data processing"]
  node_file_io["Logger file I/O<br/>[file_io.py]"]
  node_file_handler["File conditioning<br/>[file_handler.py]"]
  node_file_concat["File concatenation"]
  node_eddy_concat["EddyPro concatenation"]
  node_parser["Site data parser<br/>[data_parser.py]"]
  node_met["Meteorological calculations<br/>[met_functions.py]"]
  node_logger["Logger utilities"]
end

subgraph group_site["Site configuration"]
  node_paths["Path manager<br/>[paths_manager.py]"]
  node_site_details["Site details"]
  node_config["Config generation"]
  node_file_map["File and variable maps<br/>[data_mapper.py]"]
  node_metadata_manager["Metadata manager"]
end

subgraph group_products["Derived products"]
  node_constructors["Data constructors"]
  node_process10["10 Hz processing"]
  node_network["Data status reporting"]
  node_metadata["Metadata products"]
  node_rtmc_parser["RTMC XML parser<br/>[rtmc_xml_parser.py]"]
  node_rtmc_writer["RTMC XML writer<br/>[rtmc_xml_writer.py]"]
end

subgraph group_integration["External integrations"]
  node_transfer["Remote transfer<br/>[rclone_transfer.py]"]
  node_bom["BOM FTP access<br/>[bom_ftp.py]"]
end

node_operator(("Operator"))
node_xml["RTMC XML files"]
node_cloudstor["Cloudstor task sheet"]
node_sparql{{"TERN SPARQL service"}}
node_remote[("Remote storage")]

node_operator -->|"runs tasks"| node_tasks
node_tasks -->|"reads task list"| node_cloudstor
node_tasks -->|"runs"| node_eddy_concat
node_tasks -->|"runs"| node_constructors
node_tasks -->|"runs"| node_network
node_tasks -->|"runs"| node_process10
node_tasks -->|"runs"| node_transfer
node_tasks -->|"uses paths"| node_paths
node_parser -->|"uses file map"| node_file_map
node_parser -->|"uses handler"| node_file_handler
node_parser -->|"uses I/O"| node_file_io
node_parser -->|"uses calculations"| node_met
node_parser -->|"uses site details"| node_site_details
node_file_handler -->|"uses I/O"| node_file_io
node_file_handler -->|"uses concatenation"| node_file_concat
node_constructors -->|"parses data"| node_parser
node_constructors -->|"handles files"| node_file_handler
node_constructors -->|"uses mappings"| node_file_map
node_constructors -->|"uses paths"| node_paths
node_constructors -->|"gets site details"| node_site_details
node_network -->|"gets status data"| node_parser
node_network -->|"uses mappings"| node_file_map
node_network -->|"uses paths"| node_paths
node_network -->|"uses site details"| node_site_details
node_metadata_manager -->|"creates paths"| node_paths
node_metadata_manager -->|"creates managers"| node_file_map
node_metadata_manager -->|"gets site details"| node_site_details
node_metadata -->|"uses I/O"| node_file_io
node_metadata -->|"uses site details"| node_site_details
node_rtmc_writer -->|"uses metadata"| node_metadata_manager
node_rtmc_writer -->|"uses mappings"| node_file_map
node_rtmc_writer -->|"uses XML parser"| node_rtmc_parser
node_rtmc_writer -->|"uses paths"| node_paths
node_rtmc_parser -.->|"reads and edits"| node_xml
node_site_details -->|"queries"| node_sparql
node_transfer -->|"gets paths"| node_paths
node_config -->|"uses paths"| node_paths
node_config -->|"uses site details"| node_site_details
node_config -->|"uses I/O"| node_file_io
node_transfer -.->|"transfers files"| node_remote

click node_tasks "https://github.com/ternaustralia/vm_data_acquisition/blob/master/tasks_manager.py"
click node_paths "https://github.com/ternaustralia/vm_data_acquisition/blob/master/paths_manager.py"
click node_site_details "https://github.com/ternaustralia/vm_data_acquisition/blob/master/sparql_site_details.py"
click node_config "https://github.com/ternaustralia/vm_data_acquisition/blob/master/config_generators.py"
click node_file_map "https://github.com/ternaustralia/vm_data_acquisition/blob/master/data_mapper.py"
click node_file_io "https://github.com/ternaustralia/vm_data_acquisition/blob/master/file_io.py"
click node_file_handler "https://github.com/ternaustralia/vm_data_acquisition/blob/master/file_handler.py"
click node_file_concat "https://github.com/ternaustralia/vm_data_acquisition/blob/master/file_concatenators.py"
click node_eddy_concat "https://github.com/ternaustralia/vm_data_acquisition/blob/master/eddy_pro_concatenator.py"
click node_parser "https://github.com/ternaustralia/vm_data_acquisition/blob/master/data_parser.py"
click node_met "https://github.com/ternaustralia/vm_data_acquisition/blob/master/met_functions.py"
click node_constructors "https://github.com/ternaustralia/vm_data_acquisition/blob/master/file_constructors.py"
click node_process10 "https://github.com/ternaustralia/vm_data_acquisition/blob/master/process_10hz_data.py"
click node_network "https://github.com/ternaustralia/vm_data_acquisition/blob/master/network_status_parser.py"
click node_metadata "https://github.com/ternaustralia/vm_data_acquisition/blob/master/metadata_handlers.py"
click node_metadata_manager "https://github.com/ternaustralia/vm_data_acquisition/blob/master/metadata_handler.py"
click node_rtmc_parser "https://github.com/ternaustralia/vm_data_acquisition/blob/master/rtmc_xml_parser.py"
click node_rtmc_writer "https://github.com/ternaustralia/vm_data_acquisition/blob/master/rtmc_xml_writer.py"
click node_transfer "https://github.com/ternaustralia/vm_data_acquisition/blob/master/rclone_transfer.py"
click node_bom "https://github.com/ternaustralia/vm_data_acquisition/blob/master/bom_ftp.py"
click node_logger "https://github.com/ternaustralia/vm_data_acquisition/blob/master/logger_functions.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tasks toneBlue
class node_file_io,node_file_handler,node_file_concat,node_eddy_concat,node_parser,node_met,node_logger,node_remote toneAmber
class node_paths,node_site_details,node_config,node_file_map,node_metadata_manager toneMint
class node_constructors,node_process10,node_network,node_metadata,node_rtmc_parser,node_rtmc_writer,node_cloudstor toneRose
class node_transfer,node_bom,node_operator,node_xml,node_sparql toneIndigo
```

## Components

### Task orchestration

- [Task manager](https://github.com/ternaustralia/vm_data_acquisition/blob/master/tasks_manager.py): `tasks_manager.py`

### Logger data processing

- [Logger file I/O](https://github.com/ternaustralia/vm_data_acquisition/blob/master/file_io.py): `file_io.py`
- [File conditioning](https://github.com/ternaustralia/vm_data_acquisition/blob/master/file_handler.py): `file_handler.py`
- [File concatenation](https://github.com/ternaustralia/vm_data_acquisition/blob/master/file_concatenators.py): `file_concatenators.py`
- [EddyPro concatenation](https://github.com/ternaustralia/vm_data_acquisition/blob/master/eddy_pro_concatenator.py): `eddy_pro_concatenator.py`
- [Site data parser](https://github.com/ternaustralia/vm_data_acquisition/blob/master/data_parser.py): `data_parser.py`
- [Meteorological calculations](https://github.com/ternaustralia/vm_data_acquisition/blob/master/met_functions.py): `met_functions.py`
- [Logger utilities](https://github.com/ternaustralia/vm_data_acquisition/blob/master/logger_functions.py): `logger_functions.py`

### Site configuration

- [Path manager](https://github.com/ternaustralia/vm_data_acquisition/blob/master/paths_manager.py): `paths_manager.py`
- [Site details](https://github.com/ternaustralia/vm_data_acquisition/blob/master/sparql_site_details.py): `sparql_site_details.py`
- [Config generation](https://github.com/ternaustralia/vm_data_acquisition/blob/master/config_generators.py): `config_generators.py`
- [File and variable maps](https://github.com/ternaustralia/vm_data_acquisition/blob/master/data_mapper.py): `data_mapper.py`
- [Metadata manager](https://github.com/ternaustralia/vm_data_acquisition/blob/master/metadata_handler.py): `metadata_handler.py`

### Derived products

- [Data constructors](https://github.com/ternaustralia/vm_data_acquisition/blob/master/file_constructors.py): `file_constructors.py`
- [10 Hz processing](https://github.com/ternaustralia/vm_data_acquisition/blob/master/process_10hz_data.py): `process_10hz_data.py`
- [Data status reporting](https://github.com/ternaustralia/vm_data_acquisition/blob/master/network_status_parser.py): `network_status_parser.py`
- [Metadata products](https://github.com/ternaustralia/vm_data_acquisition/blob/master/metadata_handlers.py): `metadata_handlers.py`
- [RTMC XML parser](https://github.com/ternaustralia/vm_data_acquisition/blob/master/rtmc_xml_parser.py): `rtmc_xml_parser.py`
- [RTMC XML writer](https://github.com/ternaustralia/vm_data_acquisition/blob/master/rtmc_xml_writer.py): `rtmc_xml_writer.py`

### External integrations

- [Remote transfer](https://github.com/ternaustralia/vm_data_acquisition/blob/master/rclone_transfer.py): `rclone_transfer.py`
- [BOM FTP access](https://github.com/ternaustralia/vm_data_acquisition/blob/master/bom_ftp.py): `bom_ftp.py`

### Other components

- **Operator**
- **RTMC XML files**
- **Cloudstor task sheet**
- **TERN SPARQL service**
- **Remote storage**

## Connections

- Operator → Task manager: runs tasks (evidence: `tasks_manager.py`)
- Task manager → Cloudstor task sheet: reads task list (evidence: `tasks_manager.py`)
- Task manager → EddyPro concatenation: runs (evidence: `tasks_manager.py`)
- Task manager → Data constructors: runs (evidence: `tasks_manager.py`)
- Task manager → Data status reporting: runs (evidence: `tasks_manager.py`)
- Task manager → 10 Hz processing: runs (evidence: `tasks_manager.py`)
- Task manager → Remote transfer: runs (evidence: `tasks_manager.py`)
- Task manager → Path manager: uses paths (evidence: `tasks_manager.py`)
- Site data parser → File and variable maps: uses file map (evidence: `data_parser.py`)
- Site data parser → File conditioning: uses handler (evidence: `data_parser.py`)
- Site data parser → Logger file I/O: uses I/O (evidence: `data_parser.py`)
- Site data parser → Meteorological calculations: uses calculations (evidence: `data_parser.py`)
- Site data parser → Site details: uses site details (evidence: `data_parser.py`)
- File conditioning → Logger file I/O: uses I/O (evidence: `file_handler.py`)
- File conditioning → File concatenation: uses concatenation (evidence: `file_handler.py`)
- Data constructors → Site data parser: parses data (evidence: `file_constructors.py`)
- Data constructors → File conditioning: handles files (evidence: `file_constructors.py`)
- Data constructors → File and variable maps: uses mappings (evidence: `file_constructors.py`)
- Data constructors → Path manager: uses paths (evidence: `file_constructors.py`)
- Data constructors → Site details: gets site details (evidence: `file_constructors.py`)
- Data status reporting → Site data parser: gets status data (evidence: `network_status_parser.py`)
- Data status reporting → File and variable maps: uses mappings (evidence: `network_status_parser.py`)
- Data status reporting → Path manager: uses paths (evidence: `network_status_parser.py`)
- Data status reporting → Site details: uses site details (evidence: `network_status_parser.py`)
- Metadata manager → Path manager: creates paths (evidence: `metadata_handler.py`)
- Metadata manager → File and variable maps: creates managers (evidence: `metadata_handler.py`)
- Metadata manager → Site details: gets site details (evidence: `metadata_handler.py`)
- Metadata products → Logger file I/O: uses I/O (evidence: `metadata_handlers.py`)
- Metadata products → Site details: uses site details (evidence: `metadata_handlers.py`)
- RTMC XML writer → Metadata manager: uses metadata (evidence: `rtmc_xml_writer.py`)
- RTMC XML writer → File and variable maps: uses mappings (evidence: `rtmc_xml_writer.py`)
- RTMC XML writer → RTMC XML parser: uses XML parser (evidence: `rtmc_xml_writer.py`)
- RTMC XML writer → Path manager: uses paths (evidence: `rtmc_xml_writer.py`)
- RTMC XML parser → RTMC XML files: reads and edits
- Site details → TERN SPARQL service: queries (evidence: `sparql_site_details.py`)
- Remote transfer → Path manager: gets paths (evidence: `rclone_transfer.py`)
- Config generation → Path manager: uses paths (evidence: `config_generators.py`)
- Config generation → Site details: uses site details (evidence: `config_generators.py`)
- Config generation → Logger file I/O: uses I/O (evidence: `config_generators.py`)
- Remote transfer → Remote storage: transfers files
