How to create a matrix using memory of existing c++ memory

[GitRjc]

I have huge data sets. So I don't want to copy existing 2D (type **) or 3D (type ***) c++ arrays to a matrix. I want to create a matrix using existing memory already allocated. How to do this?

[davisking]

Use mat(): http://dlib.net/linear_algebra.html#mat

[facug91]

I performed some tests on this, and I think it could be done more efficiently, by adding some new functions in matrix.

For testing purposes only, I added the following method to dlib/matrix/matrix.h, inside matrix class:

void assign_ptr(T* ptr, int rows, int cols) {
	data.set_size(rows, cols);
	T* d = &data(0,0);
	T* aux = ptr;
	for (int i=0; i<rows*cols; i++)
		*d++ = *aux++;
}

I ran this on my computer, with an I5-9400, using Docker image nvidia/cuda:11.6.2-cudnn8-devel-ubuntu20.04, with this script:

#include <opencv2/opencv.hpp>
#include <dlib/matrix.h>
#include <dlib/opencv.h>
#include <dlib/image_transforms/assign_image.h>

int main(int argc, char** argv) {

	cv::VideoCapture cap("/* path to some video */");
	cv::Mat frame, gray;
	long long ans = 0, sumTime = 0, counter = 0;

	cap >> frame;
	dlib::matrix<unsigned char> aux(frame.rows, frame.cols);
	while (!frame.empty()) {
		cv::cvtColor(frame, gray, cv::COLOR_BGR2GRAY);

		auto t1 = std::chrono::high_resolution_clock::now();
		// uncomment the option you want to test
		dlib::assign_image<dlib::matrix<unsigned char>, dlib::cv_image<unsigned char>>(aux, gray);
		//aux = dlib::mat(gray.data, gray.rows, gray.cols);
		//aux.assign_ptr(gray.data, gray.rows, gray.cols);
		auto t2 = std::chrono::high_resolution_clock::now();
		sumTime += std::chrono::duration_cast<std::chrono::microseconds>(t2 - t1).count();

		ans += dlib::sum(aux);
		cap >> frame;
		counter++;
		if (counter >= 1000) break;
	}
	std::cout << "Avg time: " << sumTime / counter << std::endl;
	std::cout << "Sum of mats " << ans << std::endl;

	return 0;
}

With HD (1280x720) videos, this was the result (after multiple executions):

• Using mat: 1649 us avg.
• Using assign_image and cv_image: 1427 us avg.
• Using assign_ptr: 481 us avg.
  ~71% faster than the first one, and ~66% faster than the second one.

With Full HD (1920x1080) videos, this was the result (after multiple executions):

• Using mat: 3705 us avg.
• Using assign_image and cv_image: 3696 us avg.
• Using assign_ptr: 756 us avg.
  ~80% faster than the others.

After looking at these results, I think it would be a good idea to add something like this to matrix. Saving 1-3 milliseconds per frame is a lot of time in real-time applications. If you want, I could do a PR and we can discuss more correct ways to implement something like this, with checks on arrays for size, function names and whatever else is needed.

Anyway, I also thought about the possibility that I'm misusing the two existing ways of copying the frame. If that's the case, please help me fix it, and I can run the tests again.

[davisking]

Yeah that makes sense. Thanks for pointing it out. The thing to do would be to add an overloaded operator= to matrix for matrix_op<op_pointer_to_mat<T>> types, which is the type returned by the version of mat() you are using. Then it's all nice and tidy and automatic. A PR with that would be awesome :)

[facug91]

Here is the PR. Thanks Davis :)

[pfeatherstone]

I'm not quite sure why mat() is slower.

[pfeatherstone]

Yeah, i've done some tests on my machine and mat() has the same performance as your suggested change.

[facug91]

For anyone interested, this discussion continued here.

[pfeatherstone]

@facug91 You don't have to use cv::VideoCapture anymore. You can use dlib::ffmpeg::demuxer if you want. Check it out, you can find it in the dlib/media.h header.

[facug91]

Yes, and I did tried it, but I had some troubles with my videos, which apparently are recorded in yuv, and one of the examples I found wasn't working. I guess I need to read more about it, but I wanted to try out this particular issue quickly.
Anyway, I'll definitly check it out later, it's a huge job what you've done there. Have you tested the demuxer with nvdec, in order to decode in gpu?

[pfeatherstone]

Yes, and I did tried it, but I had some troubles with my videos, which apparently are recorded in yuv, and one of the examples I found wasn't working. I guess I need to read more about it, but I wanted to try out this particular issue quickly.

Anyway, I'll definitly check it out later, it's a huge job what you've done there. Have you tested the demuxer with nvdec, in order to decode in gpu?

Please submit an issue with a reproducible example.

[facug91]

I don't think it was a bug, more like my misuse. But if I see any bugs or anything, I'll do it, don't worry.
Thank you :)

[pfeatherstone]

FYI, you can use something like this:

#include <dlib/matrix.h>
#include <dlib/media.h>

using namespace dlib;
using namespace dlib::ffmpeg;

int main(int argc, const char** argv)
{
    demuxer cap({argv[1], video_enabled, audio_disabled});

    matrix<unsigned char> gray;
    while (cap.read(gray))
    {
        /*do something with gray*/
    }
}

[pfeatherstone]

Also note, in some cases, opencv is faster than ffmpeg. It depends on the video and its codec. For example, the native H264 decoder in opencv (which doesn't use ffmpeg) is really fast. However, I've noticed that updating ffmpeg to latest release usually gives really good performance boost. Feedback is welcome.

[ianwong1999]

Hi I stumbled upon a similar problem. I am trying to use svm_multiclass_linear_trainer, but I have to convert my original data (which is an arma::mat) into a std::vectordlib::matrix. I had tried mat() but seems it's not working (I guess it's because it has to be casted as an actual matrix anyway). Would there be any solution to this?

[davisking]

You have to copy your data into a dlib::matrix.