Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Merged by Bors] - Allow enabling secure mode with Kerberos #334

Closed
wants to merge 107 commits into from
Closed

[Merged by Bors] - Allow enabling secure mode with Kerberos #334

wants to merge 107 commits into from

Commits on Feb 14, 2023

  1. WIP: rekerberize 

    see #154
    @nightkr
    nightkr committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    6f6ec49 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into spike/security2

    @nightkr
    nightkr committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    7c18711 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2023

  1. Workingish but incredibly hacky Kerberized HDFS

    @nightkr
    nightkr committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    1914ebc View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2023

  1. Merge branch 'main' into spike/security2

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    277fc61 View commit details
    Browse the repository at this point in the history

  2. Use SecretOperatorVolumeSourceBuilder to build kerberos volume

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    bd090e8 View commit details
    Browse the repository at this point in the history

  3. Add kdc and SecretClass to example

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    941ff54 View commit details
    Browse the repository at this point in the history

  4. Move ssl-server.xml and ssl-client.xml into constant

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    eb6ad9e View commit details
    Browse the repository at this point in the history

  5. WIP

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    33d6cb3 View commit details
    Browse the repository at this point in the history

  6. Only add volumes conditionally

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    8d63204 View commit details
    Browse the repository at this point in the history

  7. Use actual SecretClasses instead of hardcoding

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    b9e7e72 View commit details
    Browse the repository at this point in the history

  8. Remove HADOOP_JAAS_DEBUG=true

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    2aadadf View commit details
    Browse the repository at this point in the history

  9. Use add instead of extend

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    b77b418 View commit details
    Browse the repository at this point in the history

  10. Only add volume mounts when needed

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    1453819 View commit details
    Browse the repository at this point in the history

  11. Only write either http or https nn address into hdfs-site

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    35c6a90 View commit details
    Browse the repository at this point in the history

  12. Only mount tls certs to main containers and fix other stuff

    @sbernauer
    sbernauer committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    3a3f744 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2023

  1. Use dedicated init container to create tls bundles

    @sbernauer
    sbernauer committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    5044baf View commit details
    Browse the repository at this point in the history

  2. Move kerberos service name into separate fn

    @sbernauer
    sbernauer committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    668dec4 View commit details
    Browse the repository at this point in the history

  3. Dynamically determine principal name

    @sbernauer
    sbernauer committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    29fa168 View commit details
    Browse the repository at this point in the history

  4. Read KERBEROS_REALM from krb5.conf rather than hardcoding

    @sbernauer
    sbernauer committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    21b9233 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2023

  1. WIP checkpoint, not working. Tries to use fqdn, nn fails to connect t… 

    …o jn because of DNS problems
    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    001394d View commit details
    Browse the repository at this point in the history

  2. Swtich back to using service names as principals. Works again, only t… 

    …he datanode wait-for-namenodes init containers fails because it's stupid any requires the dfs.namenode.kerberos.principal setting (and ignores the dfs.namenode.kerberos.principal.pattern) (hdfs will start anyway)
    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    aa02167 View commit details
    Browse the repository at this point in the history

  3. Only use hdfs_name within principal. It works!

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    0edee66 View commit details
    Browse the repository at this point in the history

  4. Test with more replicas

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    b750417 View commit details
    Browse the repository at this point in the history

  5. Switch back to use fqdn principals

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    320ec00 View commit details
    Browse the repository at this point in the history

  6. fix warnings

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    129dc95 View commit details
    Browse the repository at this point in the history

  7. Only specify nn principal in wait-for-namenodes when kerberos is enabled

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    8cf7a3d View commit details
    Browse the repository at this point in the history

  8. Use operator-rs main branch

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    119f29a View commit details
    Browse the repository at this point in the history

  9. Use new image

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    e029dfc View commit details
    Browse the repository at this point in the history

  10. Changed namespace of keytab secret to default so the example is direc… 

    …tly deployable and works.
    @soenkeliebau
    soenkeliebau committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    7ca3b03 View commit details
    Browse the repository at this point in the history

  11. Specify concrete principal when connecting to NN from init container

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    fb1a5fb View commit details
    Browse the repository at this point in the history

  12. Add first basic test

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    acd49d3 View commit details
    Browse the repository at this point in the history

  13. fix typo

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    bfa6699 View commit details
    Browse the repository at this point in the history

  14. Increase test number

    @sbernauer
    sbernauer committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    c58b877 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2023

  1. Add hadoop.security.authentication=kerberos to discovery CM

    @sbernauer
    sbernauer committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    d3c7a15 View commit details
    Browse the repository at this point in the history

  2. Add a teststep to access hdfs. Get it running by adding stuff to disc… 

    …overy CM
    @sbernauer
    sbernauer committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    38f406b View commit details
    Browse the repository at this point in the history

  3. linter

    @sbernauer
    sbernauer committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    3f0cb66 View commit details
    Browse the repository at this point in the history

  4. Set hdfs log level to DEBUG in test

    @sbernauer
    sbernauer committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    a89ecd8 View commit details
    Browse the repository at this point in the history

  5. Increase log level and double number of test runs

    @sbernauer
    sbernauer committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    6cb6bd6 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2023

  1. Disable logging again

    @sbernauer
    sbernauer committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    5ca4c23 View commit details
    Browse the repository at this point in the history

  2. Test PROD.MYCORP realm

    @sbernauer
    sbernauer committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    8b5503c View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2023

  1. Fix misc stuff

    @sbernauer
    sbernauer committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    d0cb73c View commit details
    Browse the repository at this point in the history

  2. Refactor stuff into kerberos.rs

    @sbernauer
    sbernauer committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    864b4b5 View commit details
    Browse the repository at this point in the history

  3. Create create_tls_cert_bundle_init_container_and_volumes fn

    @sbernauer
    sbernauer committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    0d81588 View commit details
    Browse the repository at this point in the history

  4. Revert "Create create_tls_cert_bundle_init_container_and_volumes fn" 

    This reverts commit 0d81588.
    @sbernauer
    sbernauer committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    54f14bc View commit details
    Browse the repository at this point in the history

  5. Add AD tests

    @nightkr @sbernauer
    nightkr authored and sbernauer committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    8d43afa View commit details
    Browse the repository at this point in the history

  6. Disable activeDirectory tests

    @sbernauer
    sbernauer committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    0a8bc3c View commit details
    Browse the repository at this point in the history

  7. Improve comment

    @sbernauer
    sbernauer committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    e9ae3e9 View commit details
    Browse the repository at this point in the history

  8. Don't hard-code realm when ktiniting

    @nightkr @sbernauer
    nightkr authored and sbernauer committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    ff60fae View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2023

  1. Write hadoop.kerberos.keytab.login.autorenewal.enabled to config and … 

    …discovery CM
    @sbernauer
    sbernauer committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    3236028 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2023

  1. Remove uneeded volume in tests

    @sbernauer
    sbernauer committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    97f5dae View commit details
    Browse the repository at this point in the history

  2. Disable node principals by default

    @sbernauer
    sbernauer committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    f9eff39 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2023

  1. Add wire encryption setting

    @sbernauer
    sbernauer committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    f13a0bb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into spike/security2

    @sbernauer
    sbernauer committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    877fccc View commit details
    Browse the repository at this point in the history

  3. Adopt to new secret-op crd

    @sbernauer
    sbernauer committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    ee66453 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2023

  1. Merge remote-tracking branch 'origin/main' into spike/security2

    @sbernauer
    sbernauer committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    a2a4759 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. Fix merge mistake

    @sbernauer
    sbernauer committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    aa15c3c View commit details
    Browse the repository at this point in the history

  2. Bump tests to 23.4 and respect listenerClass

    @sbernauer
    sbernauer committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    4202cbc View commit details
    Browse the repository at this point in the history

  3. Increase assert timeout

    @sbernauer
    sbernauer committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    4836e06 View commit details
    Browse the repository at this point in the history

  4. Only run kerberos tests

    @sbernauer
    sbernauer committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    037eb1a View commit details
    Browse the repository at this point in the history

Commits on May 16, 2023

  1. Merge branch 'main' into spike/security2

    @sbernauer
    sbernauer committed May 16, 2023
    Configuration menu
    Copy the full SHA
    74a648c View commit details
    Browse the repository at this point in the history

  2. Revert example replicas to 1

    @sbernauer
    sbernauer committed May 16, 2023
    Configuration menu
    Copy the full SHA
    cfb645c View commit details
    Browse the repository at this point in the history

  3. Update rust/operator/src/container.rs 

    Co-authored-by: Natalie <teo@nullable.se>
    @sbernauer @nightkr
    sbernauer and nightkr committed May 16, 2023
    Configuration menu
    Copy the full SHA
    8e548d2 View commit details
    Browse the repository at this point in the history

  4. Update rust/operator/src/container.rs 

    Co-authored-by: Natalie <teo@nullable.se>
    @sbernauer @nightkr
    sbernauer and nightkr committed May 16, 2023
    Configuration menu
    Copy the full SHA
    87e8b26 View commit details
    Browse the repository at this point in the history

  5. Remove commented out code regarding HADOOP_POLICY_XML

    @sbernauer
    sbernauer committed May 16, 2023
    Configuration menu
    Copy the full SHA
    06ca693 View commit details
    Browse the repository at this point in the history

  6. fix format

    @sbernauer
    sbernauer committed May 16, 2023
    Configuration menu
    Copy the full SHA
    5f9d600 View commit details
    Browse the repository at this point in the history

Commits on May 19, 2023

  1. Address Arch discussion feedback

    @sbernauer
    sbernauer committed May 19, 2023
    Configuration menu
    Copy the full SHA
    0856bc5 View commit details
    Browse the repository at this point in the history

  2. Add docs

    @sbernauer
    sbernauer committed May 19, 2023
    Configuration menu
    Copy the full SHA
    5011c22 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2023

  1. Update docs/modules/hdfs/pages/usage-guide/security.adoc 

    Co-authored-by: Natalie <teo@nullable.se>
    @sbernauer @nightkr
    sbernauer and nightkr committed May 22, 2023
    Configuration menu
    Copy the full SHA
    3cd0372 View commit details
    Browse the repository at this point in the history

  2. Update docs/modules/hdfs/pages/usage-guide/security.adoc 

    Co-authored-by: Natalie <teo@nullable.se>
    @sbernauer @nightkr
    sbernauer and nightkr committed May 22, 2023
    Configuration menu
    Copy the full SHA
    6eddc49 View commit details
    Browse the repository at this point in the history

  3. Update docs/modules/hdfs/pages/usage-guide/security.adoc 

    Co-authored-by: Natalie <teo@nullable.se>
    @sbernauer @nightkr
    sbernauer and nightkr committed May 22, 2023
    Configuration menu
    Copy the full SHA
    58738de View commit details
    Browse the repository at this point in the history

  4. Apply suggestions from code review 

    Co-authored-by: Natalie <teo@nullable.se>
    @sbernauer @nightkr
    sbernauer and nightkr committed May 22, 2023
    Configuration menu
    Copy the full SHA
    dbb11bd View commit details
    Browse the repository at this point in the history

  5. Apply review comment

    @sbernauer
    sbernauer committed May 22, 2023
    Configuration menu
    Copy the full SHA
    083d02d View commit details
    Browse the repository at this point in the history

  6. Remove sentence

    @sbernauer
    sbernauer committed May 22, 2023
    Configuration menu
    Copy the full SHA
    61e08d0 View commit details
    Browse the repository at this point in the history

  7. Re-enable tests

    @sbernauer
    sbernauer committed May 22, 2023
    Configuration menu
    Copy the full SHA
    693ceb2 View commit details
    Browse the repository at this point in the history

  8. Rework CRD accoring to review feedback

    @sbernauer
    sbernauer committed May 22, 2023
    Configuration menu
    Copy the full SHA
    4a2afb9 View commit details
    Browse the repository at this point in the history

  9. Rename kerberos.kerberosSecretClass to kerberos.secretClass

    @sbernauer
    sbernauer committed May 22, 2023
    Configuration menu
    Copy the full SHA
    5396e0a View commit details
    Browse the repository at this point in the history

Commits on May 24, 2023

  1. Adress Arch meeting feedback

    @sbernauer
    sbernauer committed May 24, 2023
    Configuration menu
    Copy the full SHA
    3a50250 View commit details
    Browse the repository at this point in the history

  2. charts

    @sbernauer
    sbernauer committed May 24, 2023
    Configuration menu
    Copy the full SHA
    cdefa59 View commit details
    Browse the repository at this point in the history

  3. Re-enable all test cases

    @sbernauer
    sbernauer committed May 24, 2023
    Configuration menu
    Copy the full SHA
    8006564 View commit details
    Browse the repository at this point in the history

  4. Re-add wire encryption privacy settings

    @sbernauer
    sbernauer committed May 24, 2023
    Configuration menu
    Copy the full SHA
    dd333a6 View commit details
    Browse the repository at this point in the history

Commits on May 26, 2023

  1. fix: Only add truststore settings when https is enabled

    @sbernauer
    sbernauer committed May 26, 2023
    Configuration menu
    Copy the full SHA
    3bd3170 View commit details
    Browse the repository at this point in the history

  2. test: Switch image to nightly

    @sbernauer
    sbernauer committed May 26, 2023
    Configuration menu
    Copy the full SHA
    0842ec7 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2023

  1. Also add ssl.server.truststore.location setting

    @sbernauer
    sbernauer committed May 30, 2023
    Configuration menu
    Copy the full SHA
    aab00c7 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2023

  1. Add comment

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    2765015 View commit details
    Browse the repository at this point in the history

  2. Remove redundant set -e

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    d171093 View commit details
    Browse the repository at this point in the history

  3. We only support Kerberos for HDFS >= 3.3.x

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    effae5a View commit details
    Browse the repository at this point in the history

  4. Add chaos monkey to smoke test

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    d69b2dd View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/main' into spike/security2

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    d51f97d View commit details
    Browse the repository at this point in the history

  6. docs

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    42e2dec View commit details
    Browse the repository at this point in the history

  7. Use constants consistently

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    93e0a7c View commit details
    Browse the repository at this point in the history

  8. Simplify

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    9e484cd View commit details
    Browse the repository at this point in the history

  9. Minor improvements

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    1ffa879 View commit details
    Browse the repository at this point in the history

  10. Fix kerberos hdfs version check

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    4a2ff34 View commit details
    Browse the repository at this point in the history

  11. cleanup

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    7817bc6 View commit details
    Browse the repository at this point in the history

  12. intendation

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    0e425bc View commit details
    Browse the repository at this point in the history

  13. Sett hadoop.rpc.protection to privacy

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    701d97f View commit details
    Browse the repository at this point in the history

  14. Check exit code of zk format

    @sbernauer
    sbernauer committed May 31, 2023
    Configuration menu
    Copy the full SHA
    3b6d52d View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2023

  1. Reduce test duration by restricting chaosmonkey in smoke test

    @sbernauer
    sbernauer committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    2b5405a View commit details
    Browse the repository at this point in the history

  2. Force delte pods

    @sbernauer
    sbernauer committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    d15620b View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. Link to journalnodes bug ticket

    @sbernauer
    sbernauer committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    bfaafa4 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2023

  1. Don't share /tmp ticket cache between containers

    @sbernauer
    sbernauer committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    01d4a5a View commit details
    Browse the repository at this point in the history

  2. Removed uneeded -D dfs.namenode.kerberos.principal=$PRINCIPAL 

    This is not needed any more after changing the princial names to not contain _HOST
    @sbernauer
    sbernauer committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    8aadcde View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2023

  1. Move stacktrace into dedicated file

    @sbernauer
    sbernauer committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    277c550 View commit details
    Browse the repository at this point in the history

  2. Only kinit in init(!) contains that need a ticket

    @sbernauer
    sbernauer committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    b38c7c7 View commit details
    Browse the repository at this point in the history

  3. fix: Set KERBEROS_REALM in every container, not only the ones with kinit

    @sbernauer
    sbernauer committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    ae2fec4 View commit details
    Browse the repository at this point in the history

  4. fix: Only kinit when Kerberos is enabled ;)

    @sbernauer
    sbernauer committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    3368d1f View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2023

  1. changelog

    @sbernauer
    sbernauer committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    9291fa3 View commit details
    Browse the repository at this point in the history