Filter out items without annotations

[CourchesneA]

I am trying to remove items for which I do not have annotations in a dataset. From the documentation, I have seen that the following could help me:

datum filter -m i+a ./my-dataset

it looks like it is required to have a filter, so I can add a generic filter that matches all:

datum filter -m i+a -e '/item/image[has_data=1]' -o ./my-output-dataset --dry-run ./my-dataset

However, it always results in an empty output (dry-run returns nothing). By using datum stats I can see that this test dataset had 13 images and 10 for which there are annotations.

By removing the --mode flag, I get my whole dataset printed as XML and I can see that most of the items have annotations:

<item>
  <id>my_folder/my_image001</id>
  <subset>Test</subset>
  <image>
    <has_data>1</has_data>
   ...
  </image>
  <annotation>
    <id>0</id>
    <type>bbox</type>
    <occluded>False</occluded>
    <rotation>0.0</rotation>
    <group>0</group>
    <label>mylabel</label>
    <label_id>0</label_id>
    <x>1010.77001953125</x>
    <y>359.7699890136719</y>
    <w>61.1500244140625</w>
    <h>69.20001220703125</h>
    <area>4231.582435913384</area>
  </annotation>
</item>

I would expect the -m i+a to be able to output a valid dataset of only the items for which annotations are available

[jihyeonyi]

Hi @CourchesneA,
Thank you for the reporting the issue.

First of all, the has_data field does not indicate whether there is annotation or not. It indicates the presence of image data.
And since it belongs to the image node, using -m -i will help retrieve items where has_data=1.

To filter out data with annotations, you can use the following command:

datum filter -e '/item[annotation]'

The item[annotation] checks if there is a child named annotation within the item node.

I hope my response will be helpful.

[CourchesneA]

That is what I was looking for. Thanks !!