Error scoring Tensorflow model mtcnn.pb. The second input must be a scalar, but it has shape [1,1]

[dcostea]

System information

• OS version/distro: Win10
• .NET Version (eg., dotnet --info): .NET Core 3.1

Issue

• What did you do? I have run TensorFlow scoring using mtcnn.pb model
• What happened? I got Tensorflow.TensorflowException: The second input must be a scalar, but it has shape [1,1]
  [[Node: _clooppnet/while/add/y/_2 = Switch[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](_clooppnet/while/add/y/_1, pnet/while/LoopCond/_135)]]
• What did you expect? To get the bounding box, maybe probability and landmarks as well.

Source code / logs

Here is the mtcnn.pb model: mtcnn.zip

These are the inputs:

This is one of the outputs node (box):

        class PredictedImageData
        {
            [ColumnName("box")]
            [VectorType(1)]
            public float[] BoundingBox { get; set; }

            [ColumnName("landmarks")]
            [VectorType(1)]
            public float[] Landmarks { get; set; }

            [ColumnName("prob")]
            public float Probability { get; set; }
        }

        private struct ImageNetSettings
        {
            public const int imageHeight = 224;
            public const int imageWidth = 224;
        }

        public class ImageNetData
        {
            public string ImagePath;

            //[ImageType(ImageNetSettings.imageHeight, ImageNetSettings.imageWidth)]
            //[VectorType(1)]
            //[ColumnName("input")]
            //public Bitmap Input { get; set; }

            [VectorType(1)]
            [ColumnName("min_size")]
            public float[] MinSize;

            [VectorType(1)]
            [ColumnName("factor")]
            public float[] Factor;

            [VectorType(3)]
            [ColumnName("thresholds")]
            public float[] Thresholds;
        }

        static void Main(string[] args)
        {
            var mlContext = new MLContext(seed: 1);
            var modelLocation = "mtcnn.pb";
            var data = mlContext.Data.LoadFromEnumerable(new List<ImageNetData>());

            var pipeline = mlContext
                .Transforms.LoadImages(
                    outputColumnName: "input",
                    imageFolder: "",
                    inputColumnName: nameof(ImageNetData.ImagePath))
                .Append(mlContext
                .Transforms.ResizeImages(
                    outputColumnName: "input",
                    imageWidth: ImageNetSettings.imageWidth,
                    imageHeight: ImageNetSettings.imageHeight,
                    inputColumnName: "input"))
                .Append(mlContext.Transforms.ExtractPixels(
                    outputColumnName: "input"))
                .Append(mlContext.Model.LoadTensorFlowModel(modelLocation).ScoreTensorFlowModel(
                    inputColumnNames: new[] { "input", "thresholds", "min_size", "factor" },
                    //outputColumnNames: new[] { "box", "prob", "landmarks" },
                    outputColumnNames: new[] { "box" },
                    addBatchDimensionInput: true));

            ITransformer model = pipeline.Fit(data);

            var predictionEngine = mlContext.Model.CreatePredictionEngine<ImageNetData, PredictedImageData>(model);

            var prediction = predictionEngine.Predict(new ImageNetData
            {
                ImagePath = "anastasia3.jpg",
                //Input = (Bitmap)Image.FromFile("anastasia3.jpg"),
                MinSize = new float[] { 40F },
                Factor = new float[] { 0.709F },
                Thresholds = new float[] { 0.6F, 0.7F, 0.7F }
            });
        }

[gvashishtha]

We're unfortunately not able to reproduce this error, as we can't load your model (apparently it was compiled on a CPU that doesn't support the AVX operator).

Without being able to load your model myself, my guess is that your min_size and factor inputs are defined incorrectly. Your model appears to be expecting floats as inputs, but your InputData class defines them as float arrays.

Try changing those two to floats, and see if that solves your problem.

[dcostea]

Thank you very much, Gopal, for your answer.

[ColumnName("min_size")] public float MinSize { get; set; } [ColumnName("factor")] public float Factor { get; set; }

I had this previously, before changing to float array and decorating with VectorType(1), and the exception was:
System.ArgumentOutOfRangeException: Schema mismatch for input column 'min_size': expected vector, got Single (Parameter 'inputSchema')

Maybe I'm missing some annotation?

[frank-dong-ms-zz]

@dcostea sorry for late response, I tried your model file and code in latest ML.NET release version 1.5.2 and the model can be loaded correctly, could you please have a try and see if there is any more issue here, thanks.

[dcostea]

@dcostea sorry for late response, I tried your model file and code in latest ML.NET release version 1.5.2 and the model can be loaded correctly, could you please have a try and see if there is any more issue here, thanks.

Hi Frank

I'm able to load the model now but I have problems when trying to predict as it follows:

var prediction = predictionEngine.Predict(new ImageNetData
            {
                ImagePath = "C:\\Data\\anastasia3.jpg",
                MinSize = new float[] { 40F },
                Factor = new float[] { 0.709F },
                Thresholds = new float[] { 0.6F, 0.7F, 0.7F }
            });

If I declare the parameter as float (single), I get this error:

System.ArgumentOutOfRangeException: 'Schema mismatch for input column 'min_size': **expected vector, got Single** (Parameter 'inputSchema')'

If I declare the parameter as float[], I get this error:

System.ArgumentOutOfRangeException: 'Schema mismatch for input column 'min_size': **expected vector, got VarVector<Single>** (Parameter 'inputSchema')'

If I declare the parameter as [VectorType(1)] float[], I get this error:

Tensorflow.TensorflowException: 'The second input must be a scalar, but it has shape [1,1]
	 [[Node: pnet/while/Switch_3 = Switch[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](pnet/while/Merge_3, pnet/while/LoopCond)]]
	 [[Node: pnet/while/prob1/_77 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_402_pnet/while/prob1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_clooppnet/while/add/y/_1)]]'

It looks like now the error is inside TensorFlow, but I'm stuck now.

[frank-dong-ms-zz]

@dcostea The reason you got this error is your model is expecting float type for min_size column while in ML.NET we require all input columns be vector type(I don't know why ML.NET ask this yet, there is long history on this and I will need little more to investigate this see if we can remove this constraint from ML.NET).
At the meantime, is it possible you modify the model a little bit, change input column from float to float array? I think that will unblock yourself.

[dcostea]

@dcostea The reason you got this error is your model is expecting float type for min_size column while in ML.NET we require all input columns be vector type(I don't know why ML.NET ask this yet, there is long history on this and I will need little more to investigate this see if we can remove this constraint from ML.NET).
At the meantime, is it possible you modify the model a little bit, change input column from float to float array? I think that will unblock yourself.

It makes perfect sense.
I might have another scenario where I'm stuck because of data types, but I will create another issue.
I will try to retrain the model.

Thank you!.

[frank-dong-ms-zz]

@dcostea could you please share the image "anastasia3.jpg" used for test also share what is the predict result from python of this image? I'm working on a PR to support primitive type as input for Tensorflow model and I could use anastasia3.jpg to do some test, thanks.

[frank-dong-ms-zz]

I will keep this issue open for now to track the work of enabling primitive type as input column for tensorflow transformer. @dcostea feel free to open new issue.

[dcostea]

I will keep this issue open for now to track the work of enabling primitive type as input column for tensorflow transformer. @dcostea feel free to open new issue.

I didn't train myself mtcnn model. That's why I said, "I will try" :)

I need mtcnn model for another pet project to align a face in an image, before extracting the image features with arcface model (onnx format). I have the arcface part working well but it seems I'm not pre-processing correctly the image before feature extraction.

I got the model from here: https://github.com/blaueck/tf-mtcnn/blob/master/mtcnn.pb

I deployed the entire project here (image and model included):
https://github.com/dcostea/ConsumeTensorFlow/tree/main/ConsumeTensorFlow

Thank you.

[frank-dong-ms-zz]

Close this issue as fix has already been checked in, ML.NET will allow primitive types as input and output columns.

[dcostea]

Close this issue as fix has already been checked in, ML.NET will allow primitive types as input and output columns.

Hi @frank-dong-ms
Do you have any idea when is the next release including this fix?