include draintimeout and change node drain method. #121

shreyas-badiger · 2023-06-26T17:29:33Z

Include the timeout flag in the drain command.
Replace the kubectl-binary drain command with kubectl-library drain function.
Update packages and dependencies.
Testing
- functional testing
- unit tests
- test with 100+ nodes
- edge cases - test with blocking PDBs and other scenarios.

codecov · 2023-06-26T18:52:47Z

Codecov Report

Merging #121 (4d901ac) into master (7084174) will decrease coverage by 0.69%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master     #121      +/-   ##
==========================================
- Coverage   70.74%   70.05%   -0.69%     
==========================================
  Files          12       12              
  Lines        1234     1259      +25     
==========================================
+ Hits          873      882       +9     
- Misses        298      310      +12     
- Partials       63       67       +4

Impacted Files	Coverage Δ
pkg/service/nodes.go	`73.72% <65.57%> (-8.26%)`	⬇️
pkg/service/events.go	`81.25% <100.00%> (ø)`
pkg/service/server.go	`60.25% <100.00%> (-0.09%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

pkg/service/nodes.go

shreyas-badiger · 2023-06-27T17:50:39Z

Created the PR from the wrong branch. Converting it to Draft again.

pkg/service/nodes.go

pkg/service/events_test.go

pkg/service/nodes.go

ZihanJiang96

LGTM

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

figo · 2023-06-28T23:44:39Z

not for this PR, since we have k8s client now, could we get rid of the kubectl in the following PR?

figo

LGTM

shreyas-badiger · 2023-06-29T00:20:40Z

Verified lifecycle-manager terminating for 100+ nodes on multiple ocassions.
Note: All 106 instances were terminated within 5 mins.

╰─ bat lcm_logs| grep "setting lifecycle event as completed with result: CONTINUE" | wc -l                                                                                                         ─╯
     106
╭─    ~ ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ✔  17:07:32  ─╮
╰─ bat lcm_logs| grep "setting lifecycle event as completed with result: CONTINUE"                                                                                                                 ─╯
time="2023-06-29T00:01:22Z" level=info msg="i-0012f6d9cfe9556f3> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:01:23Z" level=info msg="i-099405c99fe257b6c> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:01:38Z" level=info msg="i-08a803043df1de54d> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:01:44Z" level=info msg="i-000748844ca712065> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:02:02Z" level=info msg="i-01a16fad153bb7cc6> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:02:09Z" level=info msg="i-01b4b6b6f705b9828> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:02:20Z" level=info msg="i-0395a1a75e885c20c> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:02:21Z" level=info msg="i-02b3e25305421e53d> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:02:32Z" level=info msg="i-018a63bf9c26f544c> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:03:02Z" level=info msg="i-0883ab1737f3ff702> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:03:21Z" level=info msg="i-02bc86e5df9298201> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:03:25Z" level=info msg="i-01440df2a536500ff> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:03:41Z" level=info msg="i-07190b65392bacbf6> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:03:48Z" level=info msg="i-0320f430834ef602b> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:03:51Z" level=info msg="i-02c13d60a0dd4c88d> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:03:59Z" level=info msg="i-021d38cf673553ec1> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:02Z" level=info msg="i-00d280256dd92ffe9> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:12Z" level=info msg="i-04e3dee84f076a1a3> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:13Z" level=info msg="i-06e769161945255ba> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:20Z" level=info msg="i-0a2c7f904b125d82a> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:21Z" level=info msg="i-07d66959a292d8ee9> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:23Z" level=info msg="i-03e636d60516b19cf> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:26Z" level=info msg="i-0bc42a80269d83f9f> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:28Z" level=info msg="i-037ac7aa29cf28b9f> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:29Z" level=info msg="i-06967863d304b5765> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:30Z" level=info msg="i-04c40862602879330> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:30Z" level=info msg="i-07ecc9b83db2f37c7> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:31Z" level=info msg="i-00bfee0d398d9f10f> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:31Z" level=info msg="i-0b02e3a43e39d305a> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:32Z" level=info msg="i-04a24d06da7054ee1> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:32Z" level=info msg="i-09165a253c0640e71> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:33Z" level=info msg="i-04167c8b34f37d77e> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:34Z" level=info msg="i-0c7bfc1e9062b89e4> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:35Z" level=info msg="i-0217fff187ab3134f> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:35Z" level=info msg="i-0b5853741157a9d86> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:35Z" level=info msg="i-042410623a44a52cd> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:36Z" level=info msg="i-0b4f7c73427c13a3d> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:37Z" level=info msg="i-086290754ecb3307f> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:38Z" level=info msg="i-0a65d72b415713728> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:38Z" level=info msg="i-0a91d2fc135a7f74a> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:39Z" level=info msg="i-0bbda9ead29f24231> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:39Z" level=info msg="i-03ce4c7a3e8b812ab> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:40Z" level=info msg="i-090bd87885f651ac2> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:40Z" level=info msg="i-06d0e4c64ec1b03de> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:41Z" level=info msg="i-0be3db61f3367491c> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:42Z" level=info msg="i-09775c052ffb7d853> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:42Z" level=info msg="i-0a2a9ad92e797a032> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:43Z" level=info msg="i-0ab079057933947ab> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:45Z" level=info msg="i-0ad589ee0a302b7dc> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:46Z" level=info msg="i-0c07afe33d1426d7d> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:46Z" level=info msg="i-011a8bd7f5ed1e70f> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:46Z" level=info msg="i-0bd36b8227f4f854e> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:46Z" level=info msg="i-0d513d17117c0c052> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:47Z" level=info msg="i-0ee0a2ab5e1cbbf75> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:47Z" level=info msg="i-03a029b611a55cd3b> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:47Z" level=info msg="i-0483d90112c8bde41> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:48Z" level=info msg="i-02ed7b4b3deb31860> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:48Z" level=info msg="i-07636e2671170cf24> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:48Z" level=info msg="i-0078d71b20bcebbf2> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:48Z" level=info msg="i-0fce7c07b424162b3> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:49Z" level=info msg="i-070ede9618a0ce5eb> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:49Z" level=info msg="i-04b8e0f3fee5f78a2> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:49Z" level=info msg="i-0a1e1dd8b773aca29> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:49Z" level=info msg="i-0ed9effb7a58ee6b1> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:50Z" level=info msg="i-044c173f243771517> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:50Z" level=info msg="i-01c24b237ab2ce9fc> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:51Z" level=info msg="i-0bb8d4eca2a799aff> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:52Z" level=info msg="i-0a9b7633070e91ee0> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:52Z" level=info msg="i-0b6225a50651ec2df> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:52Z" level=info msg="i-0107078fc40b28eec> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:53Z" level=info msg="i-03c199bc1852028b3> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:54Z" level=info msg="i-019b6af7b3d11ef29> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:54Z" level=info msg="i-0b57c3105de5a4bb2> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:54Z" level=info msg="i-0d654016b927d3872> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:55Z" level=info msg="i-09279ecdf8781b04a> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:55Z" level=info msg="i-077e912ea7444469e> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:55Z" level=info msg="i-0ca7de5e59d503084> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:56Z" level=info msg="i-06274b3014c14f7b5> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:56Z" level=info msg="i-0f8f3b5a9e1b65eb3> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:56Z" level=info msg="i-09058edfc7b4d0d46> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:57Z" level=info msg="i-0d7fefa867276bb18> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:57Z" level=info msg="i-0d054c777a207ee67> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:58Z" level=info msg="i-0edaceada17341840> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:04:58Z" level=info msg="i-0e77d42ba53761dba> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:00Z" level=info msg="i-0c326eeb8d4495bc4> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:01Z" level=info msg="i-0ad881c2b4b69ed45> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:02Z" level=info msg="i-0750629a602207475> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:03Z" level=info msg="i-0b8b94652a48f3c18> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:03Z" level=info msg="i-0bc76653c113f9f6c> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:04Z" level=info msg="i-0fb8a5c827edc28ab> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:04Z" level=info msg="i-08900633bddada0ad> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:05Z" level=info msg="i-0e7eb08331c8e2cdc> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:06Z" level=info msg="i-0c10302e66af669e0> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:07Z" level=info msg="i-06c6595a869553e68> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:09Z" level=info msg="i-04f6d8b5c0b8af907> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:10Z" level=info msg="i-00d083c107e2f33cc> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:11Z" level=info msg="i-0e21d4a4a668102c7> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:12Z" level=info msg="i-07072902593f6f52f> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:13Z" level=info msg="i-0c8f859b468d0fa28> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:13Z" level=info msg="i-07c9d3e8ed85378d3> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:14Z" level=info msg="i-0ef773684095d4b76> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:14Z" level=info msg="i-0e6b3de9570afc026> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:18Z" level=info msg="i-0dc9a3fd769cab2bb> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:26Z" level=info msg="i-0fb681ef3d6792747> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:31Z" level=info msg="i-0748b88f4dc4bfaf1> setting lifecycle event as completed with result: CONTINUE"
time="2023-06-29T00:05:35Z" level=info msg="i-0ccd62c96edba78c5> setting lifecycle event as completed with result: CONTINUE"

shreyas-badiger marked this pull request as ready for review June 27, 2023 17:19

shreyas-badiger requested a review from a team as a code owner June 27, 2023 17:19

ZihanJiang96 reviewed Jun 27, 2023

View reviewed changes

pkg/service/nodes.go Outdated Show resolved Hide resolved

shreyas-badiger marked this pull request as draft June 27, 2023 17:50

shreyas-badiger marked this pull request as ready for review June 27, 2023 21:25

shreyas-badiger changed the title ~~include draintimeout and increase context timeout for node drains~~ include draintimeout and change node drain method. Jun 27, 2023

ZihanJiang96 reviewed Jun 27, 2023

View reviewed changes

pkg/service/nodes.go Show resolved Hide resolved

akhil-rane reviewed Jun 28, 2023

View reviewed changes

pkg/service/events_test.go Show resolved Hide resolved

pkg/service/nodes.go Outdated Show resolved Hide resolved

ZihanJiang96 approved these changes Jun 28, 2023

View reviewed changes

shreyas-badiger added 15 commits June 28, 2023 16:44

debug messages

78af7ad

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

increase the context timeout

b345dcc

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

add timeout to drain command

d942331

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

remove context timeout

ca132c3

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

replace kubectl-binary drain with kubectl-library drain

21c30a3

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

undo changes to fix CI

31655ba

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

resolve package dependencies

731e327

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

update function signatures

ab84c50

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

add unit tests

998ae4f

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

add retries

00c5520

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

remove unused functions

240f07b

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

check if node is part of the cluster

dd457e6

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

remove duplicate library entry.

5eb5576

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

modify the retry logic

91ebd62

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

avoid retrying-drain message after last attempt

4d901ac

Signed-off-by: sbadiger <shreyas_badiger@intuit.com>

figo approved these changes Jun 28, 2023

View reviewed changes

shreyas-badiger merged commit 279252c into keikoproj:master Jun 29, 2023
2 of 4 checks passed

shreyas-badiger mentioned this pull request Jul 19, 2023

Release 0.6.1 #130

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

include draintimeout and change node drain method. #121

include draintimeout and change node drain method. #121

shreyas-badiger commented Jun 26, 2023 •

edited

Loading

codecov bot commented Jun 26, 2023 •

edited

Loading

shreyas-badiger commented Jun 27, 2023

ZihanJiang96 left a comment

figo commented Jun 28, 2023

figo left a comment

shreyas-badiger commented Jun 29, 2023

include draintimeout and change node drain method. #121

include draintimeout and change node drain method. #121

Conversation

shreyas-badiger commented Jun 26, 2023 • edited Loading

codecov bot commented Jun 26, 2023 • edited Loading

Codecov Report

shreyas-badiger commented Jun 27, 2023

ZihanJiang96 left a comment

Choose a reason for hiding this comment

figo commented Jun 28, 2023

figo left a comment

Choose a reason for hiding this comment

shreyas-badiger commented Jun 29, 2023

shreyas-badiger commented Jun 26, 2023 •

edited

Loading

codecov bot commented Jun 26, 2023 •

edited

Loading