Set up internal Neurobagel node at BIC neurobagel-node

[alyssadai]

• request installation of necessary dependencies/QOL things from sysadmin:
  • docker compose (ideally v2+)
  • nginx (ideally we can configure /etc/nginx - maybe there's a way to grant permissions just for this folder?
  • python3.10+
  • certbot
  • tmux
  • curl
• create central Neurobagel directory under /data/origami to store JSONLD files, graph dbs, etc. -> /data/origami/neurobagel_root
  • ensure all subdirectories/file created inside the directory have rwx permissions for neurobagel group automatically
• copy existing QPN and PPMI JSONLD files into central directory (already there from last graph update)
• set up local graph database for PD data using neurobagel/recipes -> bic_pd
  • add PPMI & QPN JSONLDs
  • add vocabulary file
• deploy API in aggregate mode for now since node is still public
• set up federation API and query tool
• register subdomains:
  • api-bic.neurobagel.org
  • federate-bic.neurobagel.org
  • query-bic.neurobagel.org
• configure NGINX for above

Need root:

• restart nginx service

• set up HTTPS

• add new subdomains to uptime robot -> maybe not for now since the node is not meant to be public?

[alyssadai]

Some issues encountered:

• the node API-graph stack needs to be explicitly launched as a non-root user, to be able to write to our shared FS (otherwise, files are written as the nobody user, which causes permission errors):

CURRENT_UID=$(id -u):$(id -g) docker compose up -d

• the f-API in a container/compose network cannot reach the n-API inside its own network using the host.docker.internal syntax

• when compose networks (for local node and federation) are spun up with default network settings, sometimes the resulting IP range used by the subnet conflicts with the one used by the network FS (/data/origami)

[alyssadai]

For reviewer

Check out:

• https://api-bic.neurobagel.org
• https://federate-bic.neurobagel.org
• https://query-bic.neurobagel.org (this is pointing to the f-API)

See new Docker/Docker Compose troubleshooting tips documented here, based on encountered problems that were resolved: https://github.com/neurobagel/documentation/wiki/Docker#troubleshooting-dockerdocker-compose-recipes

Some things learned about our own tools/machines:

On neurobagel-node and st-viateur:

(Query tool)

1. When our local_federation Compose recipe is used, the resulting query tool cannot access the f-API when API_QUERY_URL is set to localhost:<fAPI port on host>, even though this is the default recommended in https://neurobagel.org/federate/#fedenv

• also doesn't work: using the container name federation with the internal port, or host.docker.internal with the published port
• Instead, API_QUERY_URL must be set to the public IP of the machine on which the f-API is running (followed by the <fAPI port on host>), or the subdomain for the f-API if NGINX is set up on the machine

3. The latest image has not been updated following the breaking change https://github.com/neurobagel/query-tool/pull/343, and as a result cannot fetch the node list from the f-API -> solution is to switch to using nightly image
4. Fetching of diagnosis + assessment options currently failing:

• This is likely due to an issue with the latest f-API image
  • The current f-API latest version includes support for the new mixed-response structure for cohort queries (/query/), but not for /attributes/{data_element_URI} queries (which was added after we switched the default tag to nightly) - consequently, it looks like the /attributes/{data_element_URI} route in the image doesn't know how to handle responses of the now-async CRUD function which is used by both endpoints
  • This incompatibility with the query tool makes sense, but what is less good is that the API tests didn't previously clue us into this potential issue (they were still passing after [ENH] Handle partial nodes success federation-api#55, which switched the CRUD function to be asyncronous, but probably at this point /attributes/{data_element_URI} endpoint already stopped working)

5. Federated queries succeed (2xx status code in network tab), but query tool doesn't show any results

• expected as query tool doesn't know how to handle the new response JSON structure yet

(f-API)

• /attributes/{data_element_URI} endpoint not working (Internal Server Error) for latest image, due to same issue as in 3 above
  • relevant errors in logs:

   /usr/local/lib/python3.10/site-packages/httpx/_client.py:1481: RuntimeWarning: coroutine 'send_get_request' was never 
  awaited for pattern, transport in self._mounts.items():
  ...  
  File "/usr/src/./app/api/crud.py", line 159, in get_terms for term in list_of_terms[data_element_URI]:
  TypeError: 'coroutine' object is not subscriptable

On neurobagel-node only:

(UNRESOLVED)

• f-API (in either a standalone container on the default bridge network, or our local_federation recipe) cannot access services on host.docker.internal
  • seemingly, only using the external URL for the n-API (NGINX) in local_nb_nodes.json works
• host.docker.internal successfully resolves to whatever the bridge network IP address is, but services with published ports on the host still can't be reached

[surchs]

Thanks a lot @alyssadai for digging through all of these issues / debugging the docker networking problems we encountered with this deployment! This will likely be very useful for some users!

Review

Check out:

• ✅ https://api-bic.neurobagel.org/
• ❌ https://federate-bic.neurobagel.org/: 502 Bad Gateway
• ❌ https://query-bic.neurobagel.org/ (this is pointing to the f-API): 502 Bad Gateway

I think NGINX for the query and federate API is not feeling well.

Within the node API, all query endpoints work correctly - datasets are shown in protected mode as expected!

Thanks very much also for documenting these problems. I think we should start collecting some of these things that are more edge cases / problems that we are not going to build generic solutions for, and then put them in the user docs - e.g. in a FAQ or "Common Problems" section.

@alyssadai, could you please

• take another look at the query tool and f-API on the BIC node and confirm they work
• open an issue for putting your notes on docker-networking and related things in an FAQ section (not on the board yet)
• open an issue (someday label?) for the unresolved host.docker.internal issue - ideally with a minimally reproducible description. Goal isn't to fix, but to have as reference for when we encounter again in the near future

And then let's close with excitement (our Douglas friends are already inquiring when we're ready for them) -> 🧑‍🍳

[alyssadai]

Thanks Seb!

The query tool and federate API should be working now - I think you tested it the other night just as I was playing around with the .env variables and relaunching containers (which probably wasn't smart timing on my part haha).

Here are the new issues:

• Containers on neurobagel-node cannot access services on host via host.docker.internal #124
• Document Docker and Docker networking troubleshooting tips documentation#168