offline_eval_map_corloc: TypeError (expected str instance, bytes found)

[ptruka]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): see Source code / logs
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04.3 LTS (Linux 69c136cba5b6 4.4.38-rt49-tegra initial commit, simple, separated models #1 SMP PREEMPT RT Tue Jul 25 09:26:02 PDT 2017 aarch64 aarch64 aarch64 GNU/Linux)
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below):
  tf.VERSION = 1.4.0
  tf.GIT_VERSION = v1.4.0-0-gd752244
  tf.COMPILER_VERSION = v1.4.0-0-gd752244
• Python version: Python 3.5.2 [numpy (1.13.3); protobuf (3.5.0.post1)]
• Bazel version (if compiling from source): 0.8.1
• GCC/Compiler version (if compiling from source): 5.4.0 20160609
• CUDA/cuDNN version: CUDA 8, cuDNN 5
• GPU model and memory: Pascal GP106, 4GB Memory
• Exact command to reproduce:

python object_detection/metrics/offline_eval_map_corloc.py \
        --eval_dir=work/train_eval/kitti_val1 \
        --eval_config_path=validation_eval_config.pbtxt \
        --input_config_path=validation_input_config_KITTI.pbtxt

Describe the problem

Previous steps:
My aim is to calculate the AP (for the class 'car') from models of the zoo on the KITTI dataset. Therefor I used the create_kitti_tf_record.py (dataset_tools). See point A in the logs below. While generating the tfrecord, i faced the same issue as #3239. The solution SamDon87 mentioned worked for me (see A3).
For the Inference (see point B) I successfully processed the smaller val tfrecord (200 images).

Error:
After executing the offline_eval_map_corloc.py (see "Exact command to reproduce", further information at point C1 and C2 in the logs) I get the following Error message:

INFO:tensorflow:Processing file: work/inference/kitticar_val1_detections.tfrecord-00000-of-00001
INFO:tensorflow:Processed 0 images...
Traceback (most recent call last):
  File "metrics/offline_eval_map_corloc.py", line 173, in <module>
    tf.app.run(main)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "metrics/offline_eval_map_corloc.py", line 166, in main
    metrics = read_data_and_evaluate(input_config, eval_config)
  File "metrics/offline_eval_map_corloc.py", line 119, in read_data_and_evaluate
    decoded_dict = data_parser.parse(example)
  File "/drive/models/research/object_detection/metrics/tf_example_parser.py", line 149, in parse
    results_dict[key] = parser.parse(tf_example)
  File "/drive/models/research/object_detection/metrics/tf_example_parser.py", line 49, in parse
    self.field_name].HasField("bytes_list") else None
TypeError: sequence item 0: expected str instance, bytes found

What I tried
To ensure that the error is not caused by the tfrecord I generated, I used another (non kitti) tfrecord. Unfortunately, the same error occurred.

Now I'm stuck and I don't know how to solve this issue.
I am grateful for any help.

Source code / logs

A1 - Command to create tfrecord

python object_detection/dataset_tools/create_kitti_tf_record_mod.py \
        --data_dir=kitti/original/data_dir \
        --output_path=datasets/kitti/original/kitticar \
        --classes_to_use=car \
        --label_map_path=datasets/kitti/kitti_label_map.pbtxt \
        --validation_set_size=200

A2 - kitti_label_map.pbtxt

item {
  id: 1
  name: 'car'
}

A3 - line 61 to 64 of the modified create_kitti_tf_record_mod.py

tf.app.flags.DEFINE_string('classes_to_use', ['car', 'pedestrian', 'dontcare'],
                         'Which classes of bounding boxes to use. Adding the'
                         'dontcare class will remove all bboxs in the dontcare'
'regions.')

B - infer_detections command

python object_detection/inference/infer_detections.py \
  --input_tfrecord_paths=datasets/kitti/original/kitticar_val.tfrecord \
  --output_tfrecord_path=work/inference/kitticar_val1_detections.tfrecord-00000-of-00001 \
  --inference_graph=ssd_inception_v2_coco_2017_11_17/frozen_inference_graph.pb \
  --discard_image_pixels

C1 - validation_input_config_KITTI.pbtxt

label_map_path: 'datasets/kitti/kitti_label_map.pbtxt'
tf_record_input_reader: { input_path: 'work/inference/kitticar_val1_detections.tfrecord@1' }

C2 - validation_eval_config.pbtxt

metrics_set: 'pascal_voc_metrics'

[tatianashp]

Could you please try using TensorFlow 1.5 and let us know if you still see the problem?

[ptruka]

Thanks for your answer, I will try Tf 1.5 today and report.

Just a general question: Is the way, -> frozen graph from the model zoo -> create inference with the graph & eval images -> run evaluation, the propriate way to receive evaluation results from a frozen graph?

---

update: Just installed TF 1.5.0 and executed the same command. Unfortunately I receive the identical error:

INFO:tensorflow:Processing file: work/inference/kitticar_val1_detections.tfrecord-00000-of-00001
INFO:tensorflow:Processed 0 images...
Traceback (most recent call last):
  File "metrics/offline_eval_map_corloc.py", line 173, in <module>
    tf.app.run(main)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 124, in run
    _sys.exit(main(argv))
  File "metrics/offline_eval_map_corloc.py", line 166, in main
    metrics = read_data_and_evaluate(input_config, eval_config)
  File "metrics/offline_eval_map_corloc.py", line 119, in read_data_and_evaluate
    decoded_dict = data_parser.parse(example)
  File "/drive/models/research/object_detection/metrics/tf_example_parser.py", line 149, in parse
    results_dict[key] = parser.parse(tf_example)
  File "/drive/models/research/object_detection/metrics/tf_example_parser.py", line 49, in parse
    self.field_name].HasField("bytes_list") else None
TypeError: sequence item 0: expected str instance, bytes found

[ptruka]

Hello,
I just gave it another try. I found the nice raccoon repo ( https://github.com/datitran/raccoon_dataset/ ) and used these files, because I thought the error might have come from my tfrecord files.

At first I took the raccoon train.record and the graph of ssd_inception_v2_coco (directly from the modelzoo). I ran infer_detections.py and received the detection.tfrecord. Next I tried to evaluate with offline_eval_map_corloc.py but the same error (TypeError: sequence item 0: expected str instance, bytes found) showed up.

After that I replaced the ssd_inception_v2_coco with the ssd_mobilenet_v1_coco graph and repeated the steps -> the same error occurred (TypeError)

Then I ran the infer_detections.py with the raccoon train.record and the raccoon output_inference_graph.pb from the repo. ... -> offline_eval_map_corloc.py -> the same error (TypeError)

---

If anyone else has an idea about what I could try out, let me know.

[tatianashp]

@tombstone Can you take a look at this?

[ohnabe]

I had same error with Python3.5 and TF1.5. This causes tf_example.features.feature[self.field_name].bytes_list.value returns byte type instead of string type in metrics/tf_example_parser.StringParser.

So I changed tf_example_parser.StringParser below

class StringParser(data_parser.DataToNumpyParser):
  """Tensorflow Example string parser."""

  def __init__(self, field_name):
    self.field_name = field_name

  def parse(self, tf_example):
    if tf_example.features.feature[self.field_name].HasField("bytes_list"):
        result = tf_example.features.feature[self.field_name].bytes_list.value
        result = "".join([x if type(x)=='str' else x.decode('utf-8') for x in result])
    else:
        result = None
    return result

I got no errors. But this is instance solution, I think there is better solution.

[ptruka]

I tried ohnabes solution and it fixed the error for me.
Thank you very much for your support!

[tatianashp]

Great. @ohnabe thank you for your help!

[varun19299]

@ohnabe

I've used your method, except with:

result = "".join([x if type(x)=='str' else x.decode('utf-8','ignore') for x in result])

to handle bits which can't be parsed.

[ohnabe]

@varun19299 Thanks! Great!

[jiteshm17]

Thanks a lot. This worked

[sumannelli-Ib]

Hi
used the below command to print the confusion matrix
python confusionmatrix.py --detections_record=test.record --label_map=training/labelmap.pbtxt --output_path=confusion_matrix.csv

I got the confusion matrix but the rows are columns are showing as zero. Precision and recall are showing as NAN. Please verify the below screenshot to verify the issue. Thanks

[jhonjam]

hello, I am using the script (https://www.shiftedup.com/2018/10/10/confusion-matrix-in-object-detection-api-with-tensorflow) , I generated the confuction matrix, only I have two classes and the matrix has a size of 3x3, which is wrong. someone could explain me. Thank you

Confusion Matrix:
[[1. 1. 7.]
[0. 3. 3.]
[2. 1. 0.]]

category ... recall_@0.5IOU
0 Glasses ... 0.111111
1 Pen ... 0.500000

[2 rows x 3 columns]

[baglanaitu]

Hi used the below command to print the confusion matrix python confusionmatrix.py --detections_record=test.record --label_map=training/labelmap.pbtxt --output_path=confusion_matrix.csv

I got the confusion matrix but the rows are columns are showing as zero. Precision and recall are showing as NAN. Please verify the below screenshot to verify the issue. Thanks

Did you solve this issue?