Missing native libraries for M1 processors

[faberf]

I am on a mac with the new M1 processor and I get this error extracting files. I get this error both in vs code and intelliJ. I am able to run setup in cineast cli successfully. I am using the standard extraction_config.json, except for a modified input.path. Here is my dataset (trimmed cifar10 with 50 images in total):
cifar10_train.zip

Steps to reproduce:

1. Follow cineast wiki tutorial on getting started. (also see my cineast.json below)
2. Create data directory (and modify extraction_config.json to have the correct path).
3. Run cottontail-dbms
4. in the cineast-api command line interface:
5. setup --clean -e extraction_config.json
6. extract -e extraction_config.json

The suspicion is that this is related to the tensorflow java binaries which are not prebuilt for my architecture.

2023-05-04 10:21:54.531 [pool-4-thread-418] FATAL o.v.c.s.r.ExtractionPipeline - Decoding Error detected!
Exception in thread "pool-4-thread-416" 2023-05-04 10:21:54.531 [pool-4-thread-418] FATAL o.v.c.s.r.ExtractionPipeline - java.lang.NoClassDefFoundError: Could not initialize class org.tensorflow.SavedModelBundle
        at org.vitrivr.cineast.core.features.InceptionResnetV2.initializeModel(InceptionResnetV2.java:130)
        at org.vitrivr.cineast.core.features.InceptionResnetV2.getModel(InceptionResnetV2.java:61)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.initializeVisualEmbedding(VisualTextCoEmbedding.java:165)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.embedVideo(VisualTextCoEmbedding.java:224)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.processSegment(VisualTextCoEmbedding.java:94)
        at org.vitrivr.cineast.standalone.runtime.ExtractionTask.run(ExtractionTask.java:28)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.bcom.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Could not initialize class org.vitrivr.cineast.core.util.text.TextDetector_EAST
ase/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: no jnitensorflow in java.library.path: /Users/faber0002/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. [in thread "pool-5-thread-VisualTextCoEmbedding"]
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2429)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
        at java.base/java.lang.System.loadLibrary(System.java:1989)
        at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1800)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1402)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1214)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
        at org.tensorflow.internal.c_api.global.tensorflow.<clinit>(tensorflow.java:12)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:467)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1269)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1214)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1206)
        at org.tensorflow.NativeLibrary.load(NativeLibrary.java:48)
        at org.tensorflow.TensorFlow.<clinit>(TensorFlow.java:140)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:375)
        at org.tensorflow.SavedModelBundle.<clinit>(SavedModelBundle.java:651)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.initializeTextEmbedding(VisualTextCoEmbedding.java:156)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.embedText(VisualTextCoEmbedding.java:173)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.getSimilar(VisualTextCoEmbedding.java:123)
        at org.vitrivr.cineast.standalone.runtime.RetrievalTask.call(RetrievalTask.java:57)
        at org.vitrivr.cineast.standalone.runtime.RetrievalTask.call(RetrievalTask.java:1)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        ... 3 more

        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2053)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950)
        at org.vitrivr.cineast.core.util.ThreadLocalObjectCache.get(ThreadLocalObjectCache.java:19)
        at org.vitrivr.cineast.core.features.OCRSearch.processSegment(OCRSearch.java:286)
        at org.vitrivr.cineast.standalone.runtime.ExtractionTask.run(ExtractionTask.java:28)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.vitrivr.cineast.core.util.text.TextDetector_EAST
        at org.vitrivr.cineast.core.features.OCRSearch$1.load(OCRSearch.java:57)
        at org.vitrivr.cineast.core.features.OCRSearch$1.load(OCRSearch.java:1)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
        ... 9 more
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "pool-4-thread-4"]
        at nu.pattern.OpenCV$LocalLoader.getInstance(OpenCV.java:340)
        at nu.pattern.OpenCV.loadLocally(OpenCV.java:323)
        at org.vitrivr.cineast.core.util.text.TextDetector_EAST.<clinit>(TextDetector_EAST.java:25)
        ... 15 more
Exception in thread "pool-4-thread-418" java.lang.NoClassDefFoundError: Could not initialize class org.tensorflow.SavedModelBundle
        at org.vitrivr.cineast.core.features.InceptionResnetV2.initializeModel(InceptionResnetV2.java:130)
        at org.vitrivr.cineast.core.features.InceptionResnetV2.getModel(InceptionResnetV2.java:61)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.initializeVisualEmbedding(VisualTextCoEmbedding.java:165)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.embedVideo(VisualTextCoEmbedding.java:224)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.processSegment(VisualTextCoEmbedding.java:94)
        at org.vitrivr.cineast.standalone.runtime.ExtractionTask.run(ExtractionTask.java:28)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: no jnitensorflow in java.library.path: /Users/faber0002/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. [in thread "pool-5-thread-VisualTextCoEmbedding"]
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2429)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
        at java.base/java.lang.System.loadLibrary(System.java:1989)
        at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1800)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1402)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1214)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
        at org.tensorflow.internal.c_api.global.tensorflow.<clinit>(tensorflow.java:12)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:467)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1269)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1214)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1206)
        at org.tensorflow.NativeLibrary.load(NativeLibrary.java:48)
        at org.tensorflow.TensorFlow.<clinit>(TensorFlow.java:140)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:375)
        at org.tensorflow.SavedModelBundle.<clinit>(SavedModelBundle.java:651)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.initializeTextEmbedding(VisualTextCoEmbedding.java:156)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.embedText(VisualTextCoEmbedding.java:173)
        at org.vitrivr.cineast.core.features.VisualTextCoEmbedding.getSimilar(VisualTextCoEmbedding.java:123)
        at org.vitrivr.cineast.standalone.runtime.RetrievalTask.call(RetrievalTask.java:57)
        at org.vitrivr.cineast.standalone.runtime.RetrievalTask.call(RetrievalTask.java:1)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        ... 3 more

Here is my cineast.json:

{
  "database": {
    "host": "localhost",
    "selector": "COTTONTAIL",
    "writer": "COTTONTAIL",
    "port": 1865,
    "plaintext": true
  },
  "retriever": {
    "threadPoolSize": 2,
    "maxResults": 10000,
    "resultsPerModule": 5000,
    "features": {
      "semantic" : [
        {"feature": "ConceptMasksAde20k", "weight" : 1.0}
      ],
      "globalcolor": [
        {"feature": "AverageColor",  "weight": 2.3},
        {"feature": "MedianColor", "weight": 1.2},
        {"feature": "AverageFuzzyHist", "weight": 0.7}
      ],
      "localcolor": [
        {"feature": "AverageColorARP44", "weight": 0.5},
        {"feature": "AverageColorGrid8", "weight": 1.8},
        {"feature": "CLD", "weight": 1.3},
        {"feature": "MedianColorGrid8", "weight": 1.7},
        {"feature": "AverageColorRaster", "weight": 1.0}
      ],
      "edge": [
        {"feature": "EHD", "weight": 0.7},
        {"feature": "DominantEdgeGrid16", "weight": 1.4}
      ],
      "localfeatures": [
        {"feature": "HOGMirflickr25K512", "weight": 1.0}
      ],
      "localfeatures_fast": [
        {"feature": "HOGMirflickr25K256", "weight": 1.0}
      ],
      "audiofingerprint": [
        {"feature": "HPCP12Shingle", "weight": 1.0}
      ],
      "ocr": [
        {"feature": "OCRSearch", "weight": 1.0}
      ],
      "visualtextcoembedding": [
        {"feature": "VisualTextCoEmbedding", "weight": 1.0}
      ],
      "boolean": [
        {
          "feature": "CollectionBooleanRetriever", "weight": 1.0,
          "properties": {
            "entity": "cineast_segment",
            "attribute": "segmentid,objectid",
            "idCol": "segmentid"
          }
        }
      ]
    }
  },

  "decoders": {
    "VIDEO": {
      "decoder": "FFMpegVideoDecoder",
      "properties": {
        "maxFrameWidth": 640,
        "maxFrameHeight": 480
      }
    },
    "IMAGE": {
      "decoder": "DefaultImageDecoder",
      "properties": {
        "bounds": 1024
      }
    },
    "AUDIO": {
      "decoder": "FFMpegAudioDecoder",
      "properties": {
        "samplerate": 44100,
        "channels": 2
      }
    }
  },

  "extractor": {
    "threadPoolSize": 4,
    "outputLocation": "out"
  },

  "api": {
    "enableCLI": true,
    "enableWebsocket": true,
    "enableWebsocketSecure": false,
    "enableRest": true,
    "enableRestLiveDoc": true,
    "enableRestSecure": false,
    "enableLegacy": false,
    "httpPort": 4567,
    "httpsPort": 4568,
    "legacyPort": 12345,
    "threadPoolSize": 12,
    "serveContent": true,
    "serveUI": false,
    "sessionExtractionConfigLocation": "extraction_config.json",
    "uiLocation": "../vitrivr-ng/dist",
    "thumbnailLocation": "./thumbnails",
    "objectLocation": "./data/",
    "objectsFilesAreIDed": false
  },

  "monitoring":{
    "enablePrometheus": false,
    "prometheusPort": 4569
  }

}

[silvanheller]

We're trying to remedy this issue on a branch (#375), but have so far not been successful

[lucaro]

Given your stack trace, it looks to me that the problem exists exclusively with features that rely on native libraries. Those native libraries are not currently available for M1 processors, at least not in the versions we're currently using. As a workaround, until this is resolved, try removing the affected features from your extraction config.

[Spiess]

Currently there are no prebuilt TensorFlow Java binaries for macOS Aarch64 as GitHub does not yet offer macOS Aarch64 build runners. Custom builds are enabled since TensorFlow Java 0.5.0, so with some effort TensorFlow features can be used on Apple Silicon macs.

There is hope for this, as GitHub has macOS Aarch64 build runners on their roadmap for end of this year.

My apologies for not opening an issue on this sooner.

I have no experience with the OpenCV features.

[Spiess]

There are good news on this issue, GitHub actions now has free M1 runners, and the JavaCPP Presets have already been made available for macOS-Aarch64.
TensorFlow Java will probably soon follow.