How varun-l/confl-loader works
Project type and purpose
• From the file tree (main.py, sk.py, sk_advanced.py) and README, this is a small Python-based integration tool (an “open-source CLI utility”) that:
• Loads credentials from a .env file
• Uses the Atlassian Python API to fetch documents from a Confluence (or similar) instance
• Parses content (via lxml)
• Feeds documents into LangChain pipelines for processing (summaries, embeddings, etc.)
• (Planned) Stores a “last fetched” timestamp and only pulls updated documents after that time
File structure
• .env – holds API_TOKEN and any other secrets
• requirements.txt – lists langchain-community, atlassian-python-api, python-dotenv, lxml
• main.py – the CLI entrypoint / orchestrator
• sk.py – core/simple implementation of the document-fetch + process logic
• sk_advanced.py – advanced/extended pipeline (e.g., additional LangChain chains or storage features)
README for additional insights
• The README shows essential pip installs and hints at future work: tracking “when” documents were last pulled and only retrieving updated docs.
Important details to capture
• Secrets/config reading step (.env → python-dotenv)
• Distinction between fetching raw document and parsing it
• The role of LangChain modules in sk.py vs. sk_advanced.py
• Highlights current vs. future functionality
• Indicates which external service (Atlassian) and which libraries power each step