nextcloud-aio-fulltextsearch connectivity issue

[iGadget]

Glancing through the Nextcloud logs in an attempt to -unsuccesfully- fix another issue, I noticed it was being filled with these entries:

[fulltextsearch_elasticsearch] Error: Retry 0: cURL error 28: Failed to connect to nextcloud-aio-fulltextsearch port 9200 after 135083 ms: Couldn't connect to server

followed by

[fulltextsearch] Warning: platform seems down. we will update index next cron tick

Of course I searched for solutions, i.e. here, and here, but I got lost in the details which do not seem to match the approach taken by Nextcloud-AIO. There's also this issue, but I'm not running GroupFolder app so that's a dead end as well.

What I've tried / checked:

• Restarting the nextcloud-aio-fulltextsearch container (makes no difference)
• Running curl http://elastic:<redacted-token-here>@localhost:9200 inside the nextcloud-aio-fulltextsearch container. Result:

{
  "name" : "aee6b3a69a2d",
  "cluster_name" : "nextcloud-aio",
  "cluster_uuid" : "<redacted-uuid-here>",
  "version" : {
    "number" : "8.12.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "6185ba65d27469afabc9bc951cded6c17c21e3f3",
    "build_date" : "2024-02-01T13:07:13.727175297Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

So I guess this means the container itself is running just fine.

• Running curl http://elastic:<redacted-token-here>@nextcloud-aio-fulltextserach:9200 inside the nextcloud-aio-nextcloud container. Result: curl: (28) Failed to connect to nextcloud-aio-fulltextsearch port 9200 after 133440 ms: Couldn't connect to server
• Checked if nextcloud-aio-fulltextsearch resolves to an IP inside the nextcloud-aio-nextcloud container. Result: nextcloud-aio-fulltextsearch has address 172.19.0.7 (my Docker network knowledge is pretty non-existent so far -running ip a inside the nextcloud-aio-fulltextsearch container returned bash: ip: commandn not found-, but I did not change this from the defaults set by the Nextcloud AIO installation. So I'm assuming this IP is correct).
• Checked if nextcloud-aio-fulltextsearch resolves to the same IP on the host running the docker daemon. Result: my external IP. Would this be the culprit? My gut feeling says no, but if it is, to which IP should it resolve and where should I configure this?
• Checked /etc/hosts on the host running the docker daemon as per this discussion which was the primary hit when . Only localhost is mentioned there for 127.0.0.1 (but since my issue is -probably?- not related to Talk / Collabora this comes as no surprise).
• Looking at the only logfile that's present on the nextcloud-aio-fulltextsearch container (/logs/gc.log) nothing shows up when I run the curl command mentioned before inside the nextcloud-aio-nextcloud container.

What else can I try to debug this?

[iGadget]

Could it be that the (automatic) update to the latest ElasticSearch version caused this problem as described here? I know the AIO setup (still?) uses the http(s)://username:password@host:port method to connect.
Question is - how do I check if the proposed workaround is applicable? I can't find the mentioned file fulltextsearch_elasticsearch/lib/Platform/ElasticSearchPlatform.php anywhere in the nextcloud-aio-fulltextsearch container... What am I missing?

Update:
Just located the file inside the nextcloud-aio-nextcloud container here: /var/www/html/custom_apps/fulltextsearch_elasticsearch/lib/Platform/ElasticSearchPlatform.php.
Lo an behold, it contains the exact code snipped as mentioned here:

$cb = ClientBuilder::create()
                        ->setHosts($hosts)
                        ->setRetries(3);

However, I also noticed this right below that function:

/**
         * setBasicAuthentication() on ClientBuilder if available, using list of hosts
         */
        private function configureAuthentication(ClientBuilder $cb, array $hosts): void {
                foreach ($hosts as $host) {
                        $user = parse_url($host, PHP_URL_USER) ?? '';
                        $pass = parse_url($host, PHP_URL_PASS) ?? '';

                        if ($user !== '' || $pass !== '') {
                                $cb->setBasicAuthentication($user, $pass);
                                return;
                        }
                }
        }

So there seems to actually be some logic in there dealing with the 'old' user:pass approach. Question is - does it work?

Update 2:
I tried adding ->setBasicAuthentication('username', 'password') to the ElasticSearchPlatform.php file. After restarting both the nextcloud-aio-nextcloud and nextcloud-aio-fulltextsearch containers, it stil fails when trying with curl from within the nextcloud-aio-nextcloud container:
curl: (28) Failed to connect to nextcloud-aio-fulltextsearch port 9200 after 134044 ms: Couldn't connect to server

Update 3
Looking more closely at this issue I realised I had to hardcode my specific user and pass for my setup inside the ElasticSearchPlatform.php file instead of the generic 'user' and 'password'. However, after updating the file (tried with and without single quotes) and restarting both containers, the issue remains.

I'm out of ideas here :-(

[szaimen]

Hi, can you post the output of sudo docker info here?

[szaimen]

Hm... Can you try to recreate the nextcloud-aio network and check if that helps?

[iGadget]

Sure, how should I do that?

[szaimen]

Usually #2557 but it looks like you did already? Can you still try another time?

[iGadget]

Done. No difference - when using the search in Nextcloud, I do get some results but the 'wheel' keeps spinning and the log shows this:

Retry 0: cURL error 28: Failed to connect to nextcloud-aio-fulltextsearch port 9200 after 133303 ms: Couldn't connect to server 
(see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for 
http://elastic:<redacted-token-here>@nextcloud-aio-fulltextsearch:9200/nextcloud-aio/_search?from=0&size=5&_source_excludes=content

Not surprisingly, trying curl from within the nextcloud-aio-nextcloud container fails as well:
curl: (28) Failed to connect to nextcloud-aio-fulltextsearch port 9200 after 133290 ms: Couldn't connect to server

[iGadget]

I'm tempted to start over from scratch with this thing... Absolutely not looking forward to it (my last attempt took me well over a week) but I don't see how else I'm going to fix all the issues I'm running into :-(

[r100gs]

Have the same issue

Retry 0: cURL error 7: Failed to connect to nextcloud-aio-fulltextsearch port 9200 after 2268 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://elastic:0a5c61cfcdf6ea37c6c21e5a66e5e0f6842f9b9281304a2a@nextcloud-aio-fulltextsearch:9200/nextcloud-aio/_doc/files%3A135213

[Green2Matter]

I have exactly same issue as well. Recreating nexcloud-aio network doesn't help. I have had 2 networks listed: nextcloud-aio and nextcloud-aio_default. After removing both of them and creating nextcloud-aio

docker network connect nextcloud-aio nextcloud-aio-mastercontainer
Error response from daemon: could not find a network matching network mode nextcloud-aio_default: network nextcloud-aio_default not found

So, I had to recreate nextcloud-aio_default as well.
And all of this changed nothing. I still see: Retry 0: cURL error 28: Failed to connect to nextcloud-aio-fulltextsearch port 9200 after 130848 ms

[lachermeierm]

I have also had these two messages in the log file since NextCloud AIO v8.0.0 or maybe later

"Warning – fulltextsearch – platform seems down. we will update index next cron tick"

"Error – fulltextsearch_elasticsearch – Retry 0: cURL error 28: Failed to connect to nextcloud-aio-fulltextsearch port 9200 after 133728 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://elastic:34f24033d69c0c6f926da89ccc316385d69f983115480922@nextcloud-aio-fulltextsearch:9200/nextcloud-aio/_doc/files%3A19974"

Messages have not disappeared with Nextcloud AIO v8.1.0 either

Debian v12.5 als Linux-Container in Proxmox v8.1.10
4 cores, 4GB ram

[Green2Matter]

Version 8.2.1 and it's exactly the same. Will it be fixed in upcoming releases / updates?

[karthikiyengar]

Might be a complete red herring, but the magic answer for me was to bump Ubuntu from 22.04 to 24.04.

[szaimen]

Hi, can you upgrade to v9.1.0 beta and check if that fixes the problem for you? See https://github.com/nextcloud/all-in-one

[qudiqudi]

This problem still exists for me on v9.3.0. removing and adding the network didn't change anything. Any other pointers, what could improve things?

[hoggatt]

Still having the issue on Nextcloud AIO 9.4.1. The accepted answer does not resolve my issue.

[qudiqudi]

According to szaimen we need to take this to the fulltextsearch issue tracker. Please make sure to subscribe to this issue as well:
nextcloud/fulltextsearch#831