Help understanding the library for stepwise learning

[hackerfactor]

Dlib has extremely technical documentation, but I can't wrap my head around how to actually use the library.

What I'm trying to do:
I have a large amount of data for training. It cannot be pre-loaded into memory. For training, each set of data must be loaded, trained, and then unloaded before loading the next set of data.

The mlp example shows how to iteratively call the train function after each step. It loads the data, calls train once, then loads the next data.

However, the mlp example effectively says "don't use mlp, use svm instead." Unfortunately, all of the svm examples only demonstrate training when all data is pre-loaded before calling train. As mentioned, I cannot pre-load all of the data before training.

Does svm support a training loop? Effectively: while(some stopping criteria) { load data, train }
If not, then are there other options or should I find a different DNN library?
If it does, can you give me an example?

For my network, each data input is an array of 7000 bytes and it outputs a single integer in the range [0:10000]. The function f(x)=y is (hopefully) mostly linear, but ti will probably need multiple layers to learn it. While the input array has 7000 bytes, that is for a single data point. I have about 2 million data pairs(x,y) already labeled for training. (Best case, it won't need anywhere near the 2 million samples. But if f(x)=y were a simple mapping, then I wouldn't need an AI system.)

[davisking]

It really depends on the nature of your problem. The SVM solvers are much more accurate and fast and turn-key than any of the stochastic gradient descent solvers used in DNNs. So if your problem really is linear using a convex solver would be much more straightforward. And 7000 bytes two million times is only 14GB. A small amount on a modern computer.

You could always use the DNN tooling in dlib (start here http://dlib.net/dnn_introduction_ex.cpp.html) if you really want to not load all the data into memory and use some kind of DNN. I will be much slower though. Which may or may not be fine for your application, depends on your needs.

[hackerfactor]

Thank you for the reply. And no, I can't load all of the training data at the same time. I'm fine with it taking longer to train due to the delay in loading more training data as it goes.

I looked over the dnn_introduction_ex.cpp.html example. However, I don't see how to load new data between training runs. This example appears to load all training data first and then calls train once: trainer.train(training_images, training_labels); Is there a way to split this function so that I can change the training input between each iterative loop?

As far as whether my data mapping is a monotonic or not: I suspect it is, but I won't know for certain until after I try training on it. I'm expecting the system to either learn relatively quickly or diverge very quickly.

[arrufat]

To train with large datasets, you can read

• http://dlib.net/dnn_introduction2_ex.cpp.html
• http://dlib.net/dnn_imagenet_train_ex.cpp.html (to see a specific example of how this is done)

[hackerfactor]

Bingo! Perfect. Thank you @arrufat.