Companion repository for the paper "Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations", presented at the 20th International Conference on Principles of Knowledge Representation and Reasoning (KR2023) (arXiv).
This repository implements section 3.2 of the paper ("Representing Demonstrations: Episodic Memories"), including the creation of Action individuals in the SOMA ontology along with the timestamped hand and object trajectories as well as additional semantic information parsed from VR force-dynamic events.
The source code for interpreting NEEMs to candidate task sequences for execution on a robot can be found in this repository.
The core functionality is implemented in vr_neem_converter/neem_converter.py, which takes a directory containing a RobCoG VR dump and produces a set of NEEMs, one for each recorded demonstration in the dump:
python neem_converter.py input_dir output_dir config_file --episode_name="My Episode"
The config_file parameter should be the path to a JSON-formatted file containing a map from OWL IRIs to URDF filepaths for each object type in the VR environment that shows up in the VR excecution trace. An example is config/neem_converter_config.json.
Before using vr_neem_converter.py, launch KnowRob and rosprolog: roslaunch vr_neem_converter prereqs.launch.
This manual step is necessary before using VRNEEMConverter on VR data from RobCoG.
- Add
<!ENTITY srdl2-cap "http://knowrob.org/kb/srdl2-cap.owl#">
<!ENTITY srdl2-comp "http://knowrob.org/kb/srdl2-comp.owl#">to the <!DOCTYPE rdf:RDF[ section of the semantic map
2. Add
xmlns:srdl2-cap="http://knowrob.org/kb/srdl2-cap.owl#"
xmlns:srdl2-comp="http://knowrob.org/kb/srdl2-comp.owl#"to the <rdf:RDF section of the semantic map
3. Replace
<owl:imports rdf:resource="package://knowrob/owl/knowrob.owl"/>with
<owl:imports rdf:resource="http://knowrob.org/kb/knowrob.owl"/>in all OWL files.