How ai4bharat/indicxlit works
IndicXlit is a multilingual transliteration product for converting between Roman and Indic scripts, with word-level inference exposed through Python, command-line, and hosted API interfaces. Its broader workflow prepares and filters parallel datasets, trains models, generates candidates, and evaluates results; the repository also documents mining and finetuning. Runtime excerpts support the inference and API relationships shown here. Training, mining, and some evaluation components are represented by their existing directories because their source wiring was not sampled.
Main entry points: engine.py RNN engine · Create_Aksharantar_JSONL.py Aksharantar formatting · xlit_translit.py Python interface · generate_result_files.py CLI inference
Components 17
Transliteration runtime
- Python interface
inference/python/xlit_translit.py - CLI inference
inference/cli/en-indic/generate_result_files.py - HTTP API
app/api_expose.py - Server routes
app/ai4bharat/transliteration/xlit_server.py - RNN engine
app/ai4bharat/transliteration/rnn/engine.py - RNN inference
app/ai4bharat/transliteration/rnn/core.py - Transformer engine
app/ai4bharat/transliteration/transformer/base_engine.py - Transformer inference
app/ai4bharat/transliteration/transformer/custom_interactive.py
Data preparation
- Corpus processing
transliterate_corpus/data_tok_norm/working/tokenize_and_normalize.py - Aksharantar formatting
Dataset_Format/Create_Aksharantar_JSONL.py - ULCA formatting
ULCA_Format/ULCA_dataset.py - Data filtration
model_training_scripts/data_filtration/refresh_data_train_all_test_valid.py
Model training
- Training workflow
model_training_scripts
Evaluation and mining
- Candidate generation
model_training_scripts/generation/generate_result_files.py - Rescoring evaluation
model_training_scripts/evaluate/evaluate_result_with_rescore_option.py - Transliteration mining
data_mining/transliteration_mining_samanantar
Other components
- User
Connections 15
- User → Python interface: submits words
- Python interface → RNN engine: requests transliteration
- User → CLI inference: provides input
- CLI inference → RNN engine: requests inference
- User → HTTP API: sends request
- HTTP API → Server routes: gets app
- Server routes → RNN engine: routes inference
- RNN engine → RNN inference: runs inference
- Transformer engine → Transformer inference: initializes
- Corpus processing → Aksharantar formatting: prepares corpus
- Aksharantar formatting → Data filtration: formats pairs
- Data filtration → Training workflow: prepares splits
- Training workflow → Candidate generation: trained model
- Candidate generation → Rescoring evaluation: produces candidates
- Transliteration mining → Aksharantar formatting: mined pairs