The steps to Rolling Restart Hosts

When you run cmux rolling-restart-hosts, you will always the folling steps.

1. Choose Cloudera Manager to rolling restart roles (only 1).

   Choose Cloudera Manager:
   
      Cloudera Manager Description  
    - ---------------- -------------
   >1 cm1              Test_Cluster1
    2 cm2              Test_Cluster2
   

2. Choose Cluster to rolling restart cluster (only 1).

   ROLLING RESTART HOSTS
   
     Choose Cluster:
   
     Cloudera Manager  Cluster  Cluster Display Name CDH Ver Secured Hosts
     ----------------- -------- -------------------- ------- ------- -----
     cm1               cluster2 cluster2               5.8.3       Y     9
   > cm1               cluster3 cluster3               5.8.2       N     6
     cm1               cluster4 cluster4               5.8.3       Y     6
     cm1               cluster5 cluster5               5.7.0       N     5
     cm1               cluster6 cluster6               5.7.0       N     6
     cm1               cluster  cluster1               5.3.8       N     1
     cm2               -        -                          -       N     3
     cm2               cluster  cluster1               5.4.9       Y     8
   

3. Choose the ROLE TYPE to rolling restart host(s).

     ROLLING RESTART HOSTS
       * Cloudera Manager : cm1
       * Cluster          : [cluster3] cluster3
   
     Choose Hosts(s):
   
     Hostname            Role types
     ------------------- -----------------------------
     ...
     host003.kakao.cmux ZK(F),RM(A),FC,JN,NN(S),HM(S)
     host004.kakao.cmux ZK(L),JHS,JN,FC,NN(A),HM(A)
   > host005.kakao.cmux ZK(F),RM(S),BL,JN,HM(S)
     host006.kakao.cmux NM,DN,HBTS,RS
     host007.kakao.cmux NM,DN,HBTS,RS
     host008.kakao.cmux NM,DN,RS,HBTS
   > host009.kakao.cmux NM,DN,RS,HBTS
     host010.kakao.cmux NM,DN,RS,HBTS
      ...
   

4. Review your selection.

   ROLLING RESTART ROLES
   ---------------------------------------------------------------------------------
    * Cloudera Manager : cm1
    * Cluster          : [cluster3] cluster3
           CDH Version : 5.4.9
           Secured     : Y
           Zookeeper   : zk1.kakao.cmux
           hbase-tool  : hbase-manager-0.98-1.4.1.jar
    * Hosts
      host005.kakao.cmux
        ├── SERVER               REPLICATED_FOLLOWER
        ├── RESOURCEMANAGER      STANDBY
        ├── BALANCER             -
        ├── JOURNALNODE          -
        └── MASTER               STANDBY
      host009.kakao.cmux
        ├── HBASETHRIFTSERVER    -
        ├── NODEMANAGER          -
        ├── REGIONSERVER         -
        └── DATANODE             -
   
     ---------------------------------------------------------------------------------
   

   If REGIONSERVER is running on these hosts, display additional information for zookeeper leader and hbase-manager.

5. Confirm your selection.

   Are you sure you want to ROLLING RESTART on the above roles (y|n)?  y
   

6. Set seconds to sleep between each batch.

   Set SECONDS TO SLEEP between batches (>= 0 secs): 3
   

7. Set the max wait time after executing a roll RESTART command.

   Set the MAX WAIT TIME after executing the RESTART command (>= 180 secs): 180
   

8. Choose whether or not you run interactive mode.

   Do you want to proceed with INTERACTIVE MODE (y|n)? y
   

9. If REGIONSERVERs are running on these hosts:

   Turn off auto balancer and export assignment of all resgions on this HBase using hbase-manager.

   Disableig auto balancer
   └── java -jar .../hbase-manager-1.2-1.4.1.jar assign zk1.kakao.cmux balancer off  | tail -1
   Automatic balancer is turned off.
   Export assignment of all Regions to /tmp/cm1_cluster3_20170222_130129.exp
   └── java -jar .../hbase-manager-1.2-1.4.1.jar assign zk1.kakao.cmux export /tmp/cm1_cluster3_20170222_130129.exp
   Exported
   

10. Perform rolling restart roles for selected hosts.

    Repeat following steps for hosts.

    (1) Stop roles

    1. Enter a role level maintenance mode.

    2. Stop role.

    Stop [host005.kakao.cmux] yarn3-NODEMANAGER-9fbfadaba0a58469c2f1691d312d3d60
    Enter maintenance mode
    └── Maintenance owners: ["ROLE"]
    Stop yarn3-NODEMANAGER-9fbfadaba0a58469c2f1691d312d3d60
    Stopped
    ...
    Stop [host005.kakao.cmux] hdfs3-DATANODE-9fbfadaba0a58469c2f1691d312d3d60
    Enter maintenance mode
    └── Maintenance owners: ["ROLE"]
    Stop hdfs3-DATANODE-9fbfadaba0a58469c2f1691d312d3d60
    Stopped
    

    If REGIONSERVERs are running on these hosts:

    Before stop REGIONSERVER role, move all Regions on this RegionServer to others using hbase-manager.

    If Nameservices for HDFS HA are running on these hosts:

    Before stop NAMENODE role, perform failover.

    (2) Start roles

    1. Start role.

    2. Exit maintenance mode.

    Start [host005.kakao.cmux] hdfs3-DATANODE-9fbfadaba0a58469c2f1691d312d3d60
    Start hdfs3-DATANODE-9fbfadaba0a58469c2f1691d312d3d60
    Started
    Exit maintenance mode
    └── Maintenance owners: []
    ...
    Start [host005.kakao.cmux] yarn3-NODEMANAGER-9fbfadaba0a58469c2f1691d312d3d60
    Start yarn3-NODEMANAGER-9fbfadaba0a58469c2f1691d312d3d60
    Started
    Exit maintenance mode
    └── Maintenance owners: []
    

    If REGIONSERVERs are running on these hosts:

    After start REGIONSERVER role, restore Regions on this role using hbase-manager.

    If JOURNALNODEs are running on this host:

    After start JOURNALNODE role, roll the edits of an HDFS Nameservice.

11. If you have HBase service

    Turn on HBase auto balancer using hbase-manager

    Do you want to turn on auto balancer (y|n)?  y
    Enabling auto Balancer
    └── java -jar .../hbase-manager-1.2-1.4.1.jar assign zk1.kakao.cmux balancer on | tail -1
    Automatic balancer is turned on
    Finish