Basic Authentication with CaddyServer is not working for minio store

[Clemens123]

Hi, love your project, thank you very much for the effort you put in!

I'm trying to add secure authentication to the MLflow-Server and am using Caddy (https://caddyserver.com/) as a reverse proxy.
For MLflow it works fine, but the MinIO-Artifact-Store is just not accessible behind the reverse proxy, I don't know why.
The point is, I need to have access to MLflow Tracking savely online.

Do you have any ideas or suggestions?

Could be a cool addon to the project, too, for people to learn about reverse proxies.

Thanks and best regards!
Clemens

[Toumash]

Did you try using the official way of using minio + caddy from minio docs?

Whats the problem exactly? Which part does not work (artifacts upload from cli/python or download from mlflow ui?). Could you post some diagram which represents the problematic situation? How did you try to diagnose it?

About the idea of showing ready to launch service with authentication

Yeah, thats basically what the readme says (so the authentication should be baked into the project)... I need to think about adding the authentication, but in a way that will be super modular, and wont require big changes in the project to change the way its distributed.
Something like a environment variable switch in the dockercompose that will run the reverse proxy with basic auth by default/swtich - depends really how to approach it with the yt video (its the main way to discover this repository, so the demo should work)

[Clemens123]

Hi Toumash, thanks for answering!

The official way of using minio+caddy that you linked, is outdated, sadly. It is written for Caddyserver v.1.
When I'm using a Caddy-file with the following content:

my-url.com
reverse_proxy s3:9000       #"s3" is the MinIO-docker-service

opening my-url.com in the browser redirects me to the MinIO-console-UI at port 9001, but the connection is is marked "not secure" in the address-bar. When I replace 's3:9000 ' with 'mlflow:5000' for the MLflow-tracking, everything works fine and the connection is secure.
I'm wondering whether maybe the connection to port 9000 is actually secure and only the MinIO-redirect to 9001 opens a new, unsafe connection...

About the idea: I don't know if you are familiar with Caddyserver, but apart from MinIO it is very easy to get a secure basic authentication up and running! So if you think this would be well-received on youtube, you should go for it :)

[Toumash]

Im gonna see whats the problem tomorrow. In the meantime please post your docker-compose with caddyserver setup.
Maybe thats a opportunity to update official minio docs :)

[Clemens123]

Caddyfile (the 'basicauth' expression can be left out):

my-url.com {
    redir /s3 /s3/
    handle_path /s3/* {
        reverse_proxy s3:9000
    }
    redir /mlflow /mlflow/
    basicauth /mlflow/* {
	    <user_name> <encrypted_hash>           # hid private information here
    }
    handle_path /mlflow/* {
        reverse_proxy mlflow:5000
    }
    handle {
        reverse_proxy ui:8051                    # unrelated service from another docker project
    }
}

docker-compose for Caddy:

version: "3"
services:
  caddy:
    image: caddy:2-alpine
    container_name: caddy
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - /caddy/data:/data
      - /caddy/config:/config
    ports:
      - 80:80
      - 443:443
    restart: unless-stopped
networks:
  default:
    name: proxy-net

docker-compose for MLflow (I use the one from this project, just added the "proxy-net"-network to make the services accessible to Caddy):

version: '3.2'
services:
  s3:
    restart: always
    image: minio/minio:latest
    container_name: aws-s3
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - MINIO_ACCESS_KEY=${AWS_ACCESS_KEY_ID}
      - MINIO_SECRET_KEY=${AWS_SECRET_ACCESS_KEY}
    command:
      server /date --console-address ":9001"
    networks:
      - default
      - proxy-net
    volumes:
      - ./s3:/date
  db:
      restart: always
      image: mysql/mysql-server:5.7.28
      container_name: mlflow_db
      expose:
          - "3306"
      environment:
          - MYSQL_DATABASE=${MYSQL_DATABASE}
          - MYSQL_USER=${MYSQL_USER}
          - MYSQL_PASSWORD=${MYSQL_PASSWORD}
          - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      volumes:
          - ./dbdata:/var/lib/mysql
      networks:
            - default
  mlflow:
      restart: always
      container_name: tracker_mlflow
      image: tracker_ml
      build:
          context: ./mlflow
          dockerfile: Dockerfile
      ports:
          - "5000:5000"
      environment:
          - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
          - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
          - AWS_DEFAULT_REGION=${AWS_REGION}
          - MLFLOW_S3_ENDPOINT_URL=http://s3:9000
      networks:
          - proxy-net
          - default
      entrypoint: ./wait-for-it.sh db:3306 -t 90 -- mlflow server --backend-store-uri mysql+pymysql://${MYSQL_USER}:${MYSQL_PASSWORD}@db:3306/${MYSQL_DATABASE} --default-artifact-root s3://${AWS_BUCKET_NAME}/ -h 0.0.0.0

networks:
  proxy-net:
      external: true

[Clemens123]

Hi, did you have a chance to look at the issue yet? I had no success in resolving it yet and am now considering to switch to a SFTP-artifact store :(

[Toumash]

I dont get one thing: about what insecure connection you are talking about?
On my end everything is insecure, because i dont have any letsencrypt certificate and everything works just fine - see the PR #11
I can normaly access minio console, minio api + mlflow tracking server with a password from the browser.
If the green lock in the browser is the only issue then maybe look into the devtools console - maybe theres some unsafe resource on the page.
Basicall from what i've experienced the browser sometimes tells you whats the reason to not display the page as "save".
In my case its because of the badly configured ssl certificate.

I cannot run any code to connect to mlflow tracking, cause for some reason i cannot just trust my local caddy certificate (im on windows), but i dont see that something would block it

[Clemens123]

Sorry for the delayed answer, I was very busy lately.
I have a letsencrypt certificate and mlflow-tracking works fine for me, too. The problem is, that the minio-API does not respond when I try to access it through the reverse-proxy (at myurl.com/s3/). I tried both simply saving artifacts from the client and sending requests through postman (software).
Can you help me: what did you do to get a response there?

[Toumash]

I've linked the PR #11
I'm not using path-based virtual host, but domain-based instead (so not example.org/s3, but s3.example.org)

Let me know if that helps

[Clemens123]

Thanks for your implementation @Toumash, that certainly helped! :)
The only remaining question for me now is, whether the communication with the MinIO-API can be sent over https (Port 443) instead of directly over port 9000. As far as I understand it, operations on MinIO need to be authenticated (AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY) - but the communication with the MinIO-API is not encrypted and therefore vulnerable to man-in-the-middle-attacks?

[Toumash]

@Clemens123
That would be the job for a Caddy Server. I believe the only change you need to make is to change vHosts names to something other ther IP or *.localhost. Then the Caddy will automatically try to generate a letsencrypt certyficate for you. (Of course you will need a public domain to do that (so that the letsencrypt servers can authorize your cert)

Docs are clear about this: https://caddyserver.com/docs/automatic-https

Unfortunately i dont have access to any any machine where i could test this out. Can you please provice an example?
I would love to put it in the youtube tutorial, but we need to figure this stuff out to the end ;D

[Clemens123]

So you are suggesting that changing the Caddy-file to

[...]

# Minio API
my.url.com:9000 {
    handle_path /* {
        reverse_proxy s3:9000
    }
}

would work? I.e. it would process an initial API-request at port 9000 by first establishing a connection over port 443 (https) and then forwarding the request to the s3-service?

[Toumash]

@Clemens123 I mean if you have domain name routing, then you can just host the minio api to the mentioned minio.example.com:443 public address and the https would be added automatically, by running letsencrypt (used internally by caddyfile i believe)

If you already have ssl certificate you could use it without taking the api public https://caddy.community/t/custom-ssl-certificate/6981

[Clemens123]

Hi, I tried to run it today and it seems like it should work. Sadly, I got an error now even though the AWS-credentials on the client are 100% the same as in the .env-file.
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
Probably I changed something somewhere without resetting - so I cannot yet confirm whether it works :(
I probably won't find time the next weeks, so I can only look further into this afterwards.
Until then, I wish you a merry christmas and a happy new year :) Thank you for your help so far!

[Toumash]

Do you have the same error on the master branch?
Have you tried to reset the docker like this?

Watch out, it removes data from containers

docker compose down;
docker compose rm;
docker compose --build

Merry Christmas and Happy New Year ^^

         _   _
        ( >o< )                         _
      ,%%/'^`\%%.                      (@)                       _
      %%'V   V`%%      6               |-|\____________________ (@)
.     %%.     ,%%     ,~.              | |                     \|-|
|     *%%.   ,%%*     | |              |-|   Happy Holidays!    | |
|       *%%%%%*       | |      ____    |_|\___________________  |-|
=l-------------------l===l----(oooo)   (@)                    `\|_|
_______________________________|--|     ~                       (@)
================,==============|  |                              ~
============,,~'')=============|  |
-------,,~''......)------------|  |
     ('............)           |  |        ,
      (........,###            |  |        I\,
       (....,#######           | __     ,_/__ \__A
        ``###########          |/'o\,   1 @, `/ ,/    ___,
        '  ###########         |I    `-''   ,  /__,--'_,-\
           ###########         |\>--.______/  /V     /
     `    ###########          |  |   `---._  \`\==. `-.
         ##########            |  |/\,--==,.) ,\\  \==. \
          #######  ,           |  /__\    `\`-. \)  _ \=.\
        ) ,*****               |  |        (`=,`-.'/ `),`-)
       / \( /  \               |  |     __(_(  \=,`--.'
      ( {/ \ ,\ )              |  |  ,=--(__,\   i\=, `-.
       \(\6/\\//               |  |  `//'\__  `  I  \=,  `-.
    /============\             |  |   ``(__ `-._,' __ \=,   \
______________________________(elya)    (__,     ,'  `-\=,  .\
                                   \     (___,  (       )=,  ,)
====================================I  ,--`(_,--.`_..  /=,  ,/
====================================I /w_,--/w_,--.__,(==__,'
Elya Arrasmith

------------------------------------------------
https://asciiart.website/index.php?art=holiday/christmas/other

[Toumash]

@Clemens123 any update on this?

[Clemens123]

Hi @Toumash , sorry for the late answer: I'm extremely busy at the moment and don't know yet when I will find the time to look into it again, deeply sorry :( I hope maybe in 2 weeks?

[Toumash]

Closing due to inactivity. The PR will be open for anyone to get in and develop further