-
Notifications
You must be signed in to change notification settings - Fork 402
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug? http: TLS handshake error from ... local error: tls: bad record MAC #6085
Comments
Hey @benatsb sorry you're experiencing this issue. I'm brining this issue to the Fleet team. This way, the team can provide follow up questions and potential next steps to resolve the issue. |
Hey @benatsb the following "Why aren't my osquery agents connecting to Fleet?" section of the docs includes a "Common problems" section: https://fleetdm.com/docs/deploying/faq#why-arent-my-osquery-agents-connecting-to-fleet
I pulled the above from the docs because it looks like you're seeing Please let me know if these instructions don't help in successfully resolving your issue. |
@benatsb I'm going to close this issue for now. If you are still encountering issues please feel free to re-open this ticket with any new information about the problem. Thank you! |
hi, i am confronting the same problems in this thread SERVER CLIENTS Certificate von LetΒ΄sEncrypt renewed with Dehydrated Browsers: Firefox 115 ESR + Chrome 117 my client repeated this logs:
and my Server this:
Out of all Logs, my fleet client run and is showed in fleet server site, but only the hostname and serialnumber, no more. For this short time the client shine online, after go Offline an no more sucedeed. Last fetched almost 54 years ago (that is a lot of time!) If i turn the client "add host" command with --insecure, all run right. But the logs in server are still present. |
I have encountered this issue with Windows clients while setting up a testing environment based on Ubuntu 22.04 LTS and fleetdm version 4.49.2 and following (rather translating) the installation guide for CentOS. One aspect that made my deployment special was the fact that I utilized a TLS certificate issued by a particular internal certification authority belonging to a public key infrastructure dedicated to testing purposes. While I maintained proper full chain certificates and keys on the server side, I experienced these issues in the server log referring to client side TLS validation errors right after client installation and indefinitely ongoing, all whilst the clients had been registered but were displayed as "offline". Thus I took a deeper look at the installed Orbit client and found out that in the client files' root directory there is an accumulation of Base64 coded root CA certificates, called "certs.pem" and comment-titled "Bundle of CA Root Certificates" from Mozilla. This said, I made the experiment inserting my own CA certificate into this file, restarted the Orbit client and suddenly the error was no longer present in the logs and the client was being displayed as "online" in the web UI. Data could be fetched, so far no functional restrictions in terms of the free version. This said - I think that the Orbit client does not fetch any custom CA that might be installed system-wide in any valid way - thus far, I can only speculate that on Windows devices, the CA certificate being installed in the Windows machine wide cryptstore. One could speculate that this might also happen while utilizing self signed certificates. @noahtalerman I have some followup questions:
|
Was able to reproduce it - this time with a TLS certificate that should be publicy trusted through validatable intermediate CA's, however, until the root CA and all intermediate CA's were added to the certificate on the server (effectively full-chaining it) OR the client's cert.pem file, the error persisted. |
Thanks @N0rthg4t3! Heads up @xpkoala, re-opening this issue now that we have a lead on repro. |
Thanks! It's on my radar. |
Estimating at 5 as it may be hard to reproduce. |
@N0rthg4t3 Can you give details on full-chaining the server certificate? Does order matter? I'm currently using a certificate file that contains the actual server certificate, then the intermediates below it and the root cert at the very bottom. However, I'm still seeing the errors. Do I maybe need to call |
@DasFaultier When full chaining certificates order does matter, specifically where to put the server certificate and where the intermediate and root CA certificates. Depending on the system, I have been fine by adhering to the order Server Certificate > Intermediate 1 > Intermediate 2 > [..,] > Root CA certificate. And at least that is what I understand of |
@xpkoala I unassigned you so we do not miss this when we have capacity. |
Hi folks!
I performed the following tests with fake certificates and can confirm the above. TestsDummy test certificates:
They were generated using the following guide. Using fullchain in Fleet server and root CA only client side
Using fullchain in Fleet server and root+intermediate bundle client side
Using leaf cert in Fleet server and root+intermediate bundle client side
Using leaf cert + intermediate bundle in Fleet server and root CA only client side
Using leaf cert in Fleet server and root CA only client side
and client side:
This is expected to fail because fleetd/osquery doesn't know of the intermediate certificate so it requires the server to send it. Next steps
For (2) I've created #20142. |
I forgot to thank @N0rthg4t3 for your feedback here! (it helped me reproduce the issue) |
#6085 - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://fleetdm.com/docs/contributing/committing-changes#changes-files) for more information. - [X] Added/updated tests - [x] Manual QA for all new/changed functionality
The above scenarios were run with the certs provided and I received the expected success / fail states outlined in the steps. |
In a secure cloud city, |
Fleet version: 4.15.0
Operating system: Windows 11
Web browser: Edge and Chrome (latest)
π§βπ» Β Expected behavior
Build a windows agent, deploy the windows agent, connect to the Fleet server with no errors.
π₯ Β Actual behavior
Windows 11 device will connect to the Fleet server, but only after I build the agent using the "insecure" flag. The server logs show the following:
More info
Fleet server is on a fresh Ubuntu server 22.04 machine. I used certbot and the "certonly" module there to generate a LetsEncrypt certificate for the server. Copied the certificates over to the fleet installation directory at
/etc/fleetdm/
Set permissions for the .key to 600.
Running server for testing with
/etc/fleetdm/fleet serve --config /etc/fleetdm/fleet.yml
fleet.yml
Built the installer for Windows using the 4.15.0 fleetctl on the same Windows Machine with no osquery or orbit installed. Docker is installed though.
.\fleetctl.exe package --type=msi --fleet-desktop --fleet-url=https://fleettest --enroll-secret=SECRET --insecure
I tried without the "--insecure" flag but that never connected. After a reboot and installing the package with the flag it connects, but error for TLS still occurs server side.
QA notes
To QA this you will need the certificates being added here: #20390
Using fullchain in Fleet server and root CA only client side (should succeed)
fleet serve
with--server_cert ./tools/test-certs/server/fullchain.cert.pem --server_key ./tools/test-certs/server/server.key.pem
and install fleetd.--fleet-certificate ./tools/test-certs/root-ca/root-ca.cert.pem
fleetctl debug connection --fleet-certificate ./tools/test-certs/root-ca/root-ca.cert.pem https://localhost:8080
Using fullchain in Fleet server and root+intermediate bundle client side (should succeed)
fleet serve
with--server_cert ./tools/test-certs/server/fullchain.cert.pem --server_key ./tools/test-certs/server/server.key.pem
--fleet-certificate ./tools/test-certs/intermediate-ca/intermediate-and-root.cert.pem
and install fleetd.fleetctl debug connection --fleet-certificate ./tools/test-certs/intermediate-ca/intermediate-and-root.cert.pem https://localhost:8080
Using leaf cert in Fleet server and root+intermediate bundle client side (should succeed)
fleet serve
with--server_cert ./tools/test-certs/server/leaf.cert.pem --server_key ./tools/test-certs/server/server.key.pem
--fleet-certificate ./tools/test-certs/intermediate-ca/intermediate-and-root.cert.pem
and install fleetd.fleetctl debug connection --fleet-certificate ./tools/test-certs/intermediate-ca/intermediate-and-root.cert.pem https://localhost:8080
Using leaf cert + intermediate bundle in Fleet server and root CA only client side (should succeed)
fleet serve
with--server_cert ./tools/test-certs/server/leaf-and-intermediate.cert.pem --server_key ./tools/test-certs/server/server.key.pem
--fleet-certificate ./tools/test-certs/root-ca/root-ca.cert.pem
and install fleetd.fleetctl debug connection --fleet-certificate ./tools/test-certs/root-ca/root-ca.cert.pem https://localhost:8080
Using leaf cert in Fleet server and root CA only client side (should fail)
fleet serve
with--server_cert ./tools/test-certs/server/leaf.cert.pem --server_key ./tools/test-certs/server/server.key.pem
--fleet-certificate ./tools/test-certs/root-ca/root-ca.cert.pem
and install fleetd.fleetctl debug connection --fleet-certificate ./tools/test-certs/root-ca/root-ca.cert.pem https://localhost:8080
The text was updated successfully, but these errors were encountered: