diff --git a/LABELING.md b/LABELING.md new file mode 100644 index 00000000..9a7f74d3 --- /dev/null +++ b/LABELING.md @@ -0,0 +1,50 @@ + +# Labeling Guidelines + +Now that the screenshots are available, they need to be labeled. The labeling phase operates on couples of comparable screenshots. + +## Images marked as compatible - y +--- +#### Couples of images that are clearly compatible. +#### They look the same. +#### firefox\_chrome\_overlay window should nearly overlap them. +--- +

+

+

+

+

+ +## Bounding boxes marked as incompatible - n +--- +#### Couples of images which are not compatible +#### They are different. +#### Mark the parts which are logically different. +> For example, improper loading of images, missing text, different design, different languages, different selections, missing bullets or checkboxes and others are marked incompatible. +--- + +## Bounding boxes marked as different yet compatible - d +--- +#### Couples of images that are compatible, but with content differences. +#### They look different. +#### Mark the parts which are logically the same. +>Since the screenshots are taken at different times in the two browsers, there are differences which are not incompatibilities but are actually due to the different timing. For example, a banner could be showing a different advertisement, a video could be in two different frames, a clock could be showing different time, a captcha could be showing different characters or images, two news could be different, and so on. +--- +

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+ + +In the training phase, the best case is that we are able to detect between **Y + D and N**. If we are not able to do that, we should at least aim for the relaxed problem of detecting between **Y and D + N**. This is why we have this three labeling system. \ No newline at end of file diff --git a/README.md b/README.md index 1dbbc4b2..76ea6d56 100644 --- a/README.md +++ b/README.md @@ -16,39 +16,7 @@ The crawler repeats the same steps in Firefox and Chrome, generating a set of co The `data/` directory contains the screenshots generated by the crawler (N.B.: This directory is not present in the repository itself, but it will be created automatically after you setup the project as described in the **Setup** paragraph). ### Labeling - -Now that the screenshots are available, they need to be labeled. The labeling phase operates on couples of comparable screenshots. - -There are three possible labels: -1. **Y** for couples of images that are clearly compatible; -2. **D** for couples of images that are compatible, but with content differences (e.g. on a news site, two screenshots could be compatible even though they are showing two different news, simply because the news shown depends on the time the screenshot was taken and not on the fact that the browser is different); -3. **N** for couples of images which are not compatible. - -Here are some examples of the three labels: - -**Y** - - -**D** - - -**N** - - -In the training phase, the best case is that we are able to detect between Y+D and N. If we are not able to do that, we should at least aim for the relaxed problem of detecting between Y and D+N. This is why we have this three labeling system. - -The labeling technical details are described [in this issue](https://github.com/marco-c/autowebcompat/issues/2). - -The bounding-box labeling allows us to store the areas where the incompatibilities lie. - - - - - -- Press 'y' to mark the images as compatible; -- Press 'Enter' to select the regions; -- Click the 'T' button in the top left corner of a boundary box to toggle between classes. Purple corresponds to 'n', yellow corresponds to 'd'; -- Press 'Enter' to save changes. +[Labeling Guide](LABELING.md) ### Training diff --git a/labeling_guide/d1.png b/labeling_guide/d1.png new file mode 100644 index 00000000..f9357d49 Binary files /dev/null and b/labeling_guide/d1.png differ diff --git a/labeling_guide/n1.png b/labeling_guide/n1.png new file mode 100644 index 00000000..066564fe Binary files /dev/null and b/labeling_guide/n1.png differ diff --git a/labeling_guide/n10.png b/labeling_guide/n10.png new file mode 100644 index 00000000..2c85722d Binary files /dev/null and b/labeling_guide/n10.png differ diff --git a/labeling_guide/n11.png b/labeling_guide/n11.png new file mode 100644 index 00000000..3e09d5bf Binary files /dev/null and b/labeling_guide/n11.png differ diff --git a/labeling_guide/n12.png b/labeling_guide/n12.png new file mode 100644 index 00000000..4b3fb463 Binary files /dev/null and b/labeling_guide/n12.png differ diff --git a/labeling_guide/n13.png b/labeling_guide/n13.png new file mode 100644 index 00000000..43c02fc8 Binary files /dev/null and b/labeling_guide/n13.png differ diff --git a/labeling_guide/n14.png b/labeling_guide/n14.png new file mode 100644 index 00000000..07d85cfb Binary files /dev/null and b/labeling_guide/n14.png differ diff --git a/labeling_guide/n2.png b/labeling_guide/n2.png new file mode 100644 index 00000000..acccdf68 Binary files /dev/null and b/labeling_guide/n2.png differ diff --git a/labeling_guide/n3.png b/labeling_guide/n3.png new file mode 100644 index 00000000..f2e6c148 Binary files /dev/null and b/labeling_guide/n3.png differ diff --git a/labeling_guide/n4.png b/labeling_guide/n4.png new file mode 100644 index 00000000..d890ef0c Binary files /dev/null and b/labeling_guide/n4.png differ diff --git a/labeling_guide/n5.png b/labeling_guide/n5.png new file mode 100644 index 00000000..4ea2c679 Binary files /dev/null and b/labeling_guide/n5.png differ diff --git a/labeling_guide/n6.png b/labeling_guide/n6.png new file mode 100644 index 00000000..d68a6f3f Binary files /dev/null and b/labeling_guide/n6.png differ diff --git a/labeling_guide/n7.png b/labeling_guide/n7.png new file mode 100644 index 00000000..c7387b6c Binary files /dev/null and b/labeling_guide/n7.png differ diff --git a/labeling_guide/n8.png b/labeling_guide/n8.png new file mode 100644 index 00000000..c204edc6 Binary files /dev/null and b/labeling_guide/n8.png differ diff --git a/labeling_guide/n9.png b/labeling_guide/n9.png new file mode 100644 index 00000000..230bf9c9 Binary files /dev/null and b/labeling_guide/n9.png differ diff --git a/labeling_guide/y1.png b/labeling_guide/y1.png new file mode 100644 index 00000000..d47fcded Binary files /dev/null and b/labeling_guide/y1.png differ diff --git a/labeling_guide/y2.png b/labeling_guide/y2.png new file mode 100644 index 00000000..863ba125 Binary files /dev/null and b/labeling_guide/y2.png differ diff --git a/labeling_guide/y3.png b/labeling_guide/y3.png new file mode 100644 index 00000000..3c6882bb Binary files /dev/null and b/labeling_guide/y3.png differ diff --git a/labeling_guide/y4.png b/labeling_guide/y4.png new file mode 100644 index 00000000..0d2f52e2 Binary files /dev/null and b/labeling_guide/y4.png differ diff --git a/labeling_guide/y5.png b/labeling_guide/y5.png new file mode 100644 index 00000000..c90f6c2e Binary files /dev/null and b/labeling_guide/y5.png differ diff --git a/labeling_guide/y6.png b/labeling_guide/y6.png new file mode 100644 index 00000000..e85b936e Binary files /dev/null and b/labeling_guide/y6.png differ diff --git a/labeling_guide/y7.png b/labeling_guide/y7.png new file mode 100644 index 00000000..b004d2bc Binary files /dev/null and b/labeling_guide/y7.png differ