How j96w/mimicplay works
In MimicPlay the “prompting” is not a separate interactive UI element or module within the core code; rather, it’s integrated into the way you provide task specifications and visual cues to the system. This happens via two main mechanisms:
Task Specifications with BDDL
• When running data collection or evaluation, you see that a BDDL file is supplied (for example, in the command “python scripts/collect_playdata.py --bddl-file 'scripts/bddl_files/KITCHEN_SCENE9_playdata.bddl'” or in evaluation scripts).
• These BDDL files encode the task goals (for instance, “turn on stove”, “put bowl on shelf”) and essentially “prompt” the system on what the episode or demonstration should aim to accomplish.
Video Prompts as Input Cues
• When training the plan-guided low-level robot controller or during evaluation, a “video_prompt” argument is used. For example, in the training command:
“python scripts/train.py --config configs/lowlevel.json --dataset 'datasets/playdata/image_demo_local.hdf5' --bddl_file 'scripts/bddl_files/KITCHEN_SCENE9_eval-task-1_turn_on_stove_put_pan_on_stove_put_bowl_on_shelf.bddl' --video_prompt 'datasets/eval-task-1_turn_on_stove_put_pan_on_stove_put_bowl_on_shelf/image_demo.hdf5'”
• Here the video prompt file (an hdf5 dataset with visual observations) acts as the visual “hint” to guide the agent. It is a prompt telling the model what particular visual scenario or task is under consideration.
• Similarly, during evaluation with “scripts/run_trained_agent.py”, the “--video_prompt” parameter serves the same role.
Both types of inputs – the BDDL goal specification and the video prompt – serve as ways to provide external “instructions” or task cues to the system. They are defined and utilized within the scripts (located in the “mimicplay/scripts/” directory) that manage data collection, training, and evaluation.
So, to directly answer your query: the prompting happens in the command line usage of the training and evaluation scripts, where you supply the BDDL files (e.g., in “scripts/bddl_files/”) alongside the “video_prompt” parameter (pointing to a dataset file with the visual cues). These parameters are the key channels through which you “prompt” the system regarding what task to perform.
• The orchestration layer (the “scripts” module) where these command-line inputs are parsed.
• The integration of external task definitions via BDDL files.
• The role of video prompt inputs feeding into the low-level controller training and evaluation.