out: IOError: [Errno 2] No such file or directory: '/etc/cassandra/conf/cassandra.yaml'

[EmViKay]

cassandra-snapshotter --s3-bucket-name=xxxs --s3-bucket-region=us-east-1 --s3-base-path=mycluster backup --hosts=cassandra1
[cassandra1] Executing task 'node_start_backup'
[cassandra1] Executing task 'upload_node_backups'
[cassandra1] run: cassandra-snapshotter-agent create-upload-manifest --manifest_path=/tmp/backupmanifest --snapshot_name=20180430115314 --snapshot_keyspaces= --snapshot_table= --conf_path=/etc/cassandra/conf/ --exclude_tables=
[cassandra1] out: Traceback (most recent call last):
[cassandra1] out: File "/home/user/.local/bin/cassandra-snapshotter-agent", line 11, in
[cassandra1] out: sys.exit(main())
[cassandra1] out: File "/home/user/.local/lib/python2.7/site-packages/cassandra_snap/agent.py", line 314, in main
[cassandra1] out: args.incremental_backups
[cassandra1] out: File "/home/user/.local/lib/python2.7/site-packages/cassandra_snap/agent.py", line 210, in create_upload_manifest
[cassandra1] out: data_paths = get_data_path(conf_path)
[cassandra1] out: File "/home/user/.local/lib/python2.7/site-packages/cassandra_snap/agent.py", line 191, in get_data_path
[cassandra1] out: with open(config_file_path, 'r') as f:
[cassandra1] out: IOError: [Errno 2] No such file or directory: '/etc/cassandra/conf/cassandra.yaml'
[cassandra1] out:

Fatal error: run() received nonzero return code 1 while executing!

Requested: cassandra-snapshotter-agent create-upload-manifest --manifest_path=/tmp/backupmanifest --snapshot_name=20180430115314 --snapshot_keyspaces= --snapshot_table= --conf_path=/etc/cassandra/conf/ --exclude_tables=
Executed: /bin/bash -l -c "cassandra-snapshotter-agent create-upload-manifest --manifest_path=/tmp/backupmanifest --snapshot_name=20180430115314 --snapshot_keyspaces= --snapshot_table= --conf_path=/etc/cassandra/conf/ --exclude_tables="

Aborting.

Fatal error: One or more hosts failed while executing task 'upload_node_backups'

Aborting.
[cassandra1] Executing task 'clear_node_snapshot'
[cassandra1] run: /usr/bin/nodetool clearsnapshot -t "20180430115314"
[cassandra1] out: Requested clearing snapshot(s) for [all keyspaces] with snapshot name [20180430115314]
[cassandra1] out:

How can Fix that error?
My cassandra yaml in etc/cassandra by default

[EmViKay]

Copied manually yaml file.But it isn't OK.

After that ran once again.

cassandra-snapshotter --s3-bucket-name=xxx --s3-bucket-region=us-east-1 --s3-base-path=mycluster backup --hosts=cassandra1
[cassandra1] Executing task 'node_start_backup'
[cassandra1] Executing task 'upload_node_backups'
[cassandra1] run: cassandra-snapshotter-agent --incremental_backups create-upload-manifest --manifest_path=/tmp/backupmanifest --snapshot_name=20180430124647 --snapshot_keyspaces= --snapshot_table= --conf_path=/etc/cassandra/conf/ --exclude_tables=
[cassandra1] run: cassandra-snapshotter-agent --incremental_backups put --s3-bucket-name=xxx --s3-bucket-region=us-east-1 --s3-base-path=mycluster/20180430124647/cassandra1 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4
[cassandra1] out: cassandra_snap.agent INFO MSG: Initialized multipart upload for file /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1370-big-Index.db to mycluster/20180430124647/cassandra1/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1370-big-Index.db.lzo
[cassandra1] out: STRUCTURED: time=2018-04-30T13:09:24.535421-00 pid=11025
[cassandra1] out: cassandra_snap.agent INFO MSG: Initialized multipart upload for file /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1323-big-Data.db to mycluster/20180430124647/cassandra1/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1323-big-Data.db.lzo
[cassandra1] out: STRUCTURED: time=2018-04-30T13:09:24.536157-00 pid=11025
[cassandra1] out: cassandra_snap.agent INFO MSG: Initialized multipart upload for file /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1322-big-Digest.crc32 to mycluster/20180430124647/cassandra1/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1322-big-Digest.crc32.lzo
[cassandra1] out: STRUCTURED: time=2018-04-30T13:09:24.536436-00 pid=11025
[cassandra1] out: cassandra_snap.agent INFO MSG: Initialized multipart upload for file /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1326-big-Filter.db to mycluster/20180430124647/cassandra1/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1326-big-Filter.db.lzo
[cassandra1] out: STRUCTURED: time=2018-04-30T13:09:24.536748-00 pid=11025
[cassandra1] out: Traceback (most recent call last):
[cassandra1] out: File "/home/user/.local/bin/cassandra-snapshotter-agent", line 11, in
[cassandra1] out: sys.exit(main())
[cassandra1] out: File "/home/user/.local/lib/python2.7/site-packages/cassandra_snap/agent.py", line 336, in main
[cassandra1] out: args.incremental_backups
[cassandra1] out: File "/home/user/.local/lib/python2.7/site-packages/cassandra_snap/agent.py", line 182, in put_from_manifest
[cassandra1] out: os.remove(f)
[cassandra1] out: OSError: [Errno 13] Permission denied: '/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1370-big-Index.db'
[cassandra1] out: cassandra_snap.agent INFO MSG: Initialized multipart upload for file /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1302-big-TOC.txt to mycluster/20180430124647/cassandra1/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1302-big-TOC.txt.lzo
[cassandra1] out: STRUCTURED: time=2018-04-30T13:09:25.411323-00 pid=11025
[cassandra1] out: cassandra_snap.agent INFO MSG: Initialized multipart upload for file /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1322-big-Index.db to mycluster/20180430124647/cassandra1/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1322-big-Index.db.lzo
[cassandra1] out: STRUCTURED: time=2018-04-30T13:09:25.561325-00 pid=11025
[cassandra1] out: cassandra_snap.agent INFO MSG: Initialized multipart upload for file /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1332-big-Filter.db to mycluster/20180430124647/cassandra1/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups/mc-1332-big-Filter.db.lzo
[cassandra1] out: STRUCTURED: time=2018-04-30T13:09:25.562733-00 pid=11025
[cassandra1] out:

Fatal error: run() received nonzero return code 1 while executing!

Requested: cassandra-snapshotter-agent --incremental_backups put --s3-bucket-name=xxx --s3-bucket-region=us-east-1 --s3-base-path=mycluster/20180430124647/cassandra1 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4
Executed: /bin/bash -l -c "cassandra-snapshotter-agent --incremental_backups put --s3-bucket-name=xxx --s3-bucket-region=us-east-1 --s3-base-path=mycluster/20180430124647/cassandra1 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4"

Aborting.

Fatal error: One or more hosts failed while executing task 'upload_node_backups'

Aborting.

[gamunu]

Looks like a permission issue. It need write access to the data directory. If your Cassandra process is running under a different user switch to that and execute or if it's a nologin user you will have to setfactl first or use root.

[EmViKay]

i ran this command from user "user". On the node i have also user "user". /var/lib/cassandra/data right cassandra/cassandra.
Which the best way to fix?

[gamunu]

Try setfacl -R -m u:user:rwx /var/lib/data which will add an ACL entry for user user without modifying permissions.

[EmViKay]

Thx. Will try.

Also i had another problem after added second node.

cassandra-snapshotter --s3-bucket-name=xxx --s3-bucket-region=us-east-1 --s3-base-path=mycluster backup --hosts=cassandra1,cassandra2
[cassandra1] Executing task 'node_start_backup'
[cassandra2] Executing task 'node_start_backup'

Fatal error: run() received nonzero return code 1 while executing!

Requested: /usr/bin/nodetool snapshot -t 20180501083510
Executed: /bin/bash -l -c "/usr/bin/nodetool snapshot -t 20180501083510 "

================================================================================================= Standard output =================================================================================================

nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.

===================================================================================================================================================================================================================

Aborting.

Fatal error: One or more hosts failed while executing task 'node_start_backup'

Aborting.
[cassandra1] Executing task 'clear_node_snapshot'
[cassandra2] Executing task 'clear_node_snapshot'
[cassandra2] run: /usr/bin/nodetool clearsnapshot -t "20180501083510"
[cassandra1] run: /usr/bin/nodetool clearsnapshot -t "20180501083510"
[cassandra2] out: nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
[cassandra2] out:

Fatal error: run() received nonzero return code 1 while executing!

Requested: /usr/bin/nodetool clearsnapshot -t "20180501083510"
Executed: /bin/bash -l -c "/usr/bin/nodetool clearsnapshot -t "20180501083510""

Aborting.
[cassandra1] out: Requested clearing snapshot(s) for [all keyspaces] with snapshot name [20180501083510]
[cassandra1] out:

Fatal error: One or more hosts failed while executing task 'clear_node_snapshot'

[gamunu]

Yes. This is an issue with the library wich the tool is built on. It needs ssh access without a password (ssh key) for executing backup operations even in if it's the same machine. http://www.linuxproblem.org/art_9.html

[EmViKay]

Look likes problem with second node. I have configured second node with no password.
Can't start command nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
But on first node it works fine.

Thank you for your time and help.

[EmViKay]

Don't want to create another issue.
Posted question here.
Trying to restore from S3 to one of node. (deleted files manually from /var/lib/cassandra/data/keyspace/table)
Using command:
cassandra-snapshotter --s3-bucket-name=xxx --s3-bucket-region=us-east-1 --s3-base-path=mycluster restore --target-hosts=cassandra1 --keyspace test3 --snapshot-name 20180501121212

And got this:
WARNING:root:Deleting directory (/tmp/restore_cassandra/test3)...
Found 68 files, with total size of 19.6KB.
Unknown host: cassandra1: Name or service not known
usage: sstableloader [options] <dir_path>

Bulk load the sstables found in the directory <dir_path> to the configured
cluster.The parent directories of <dir_path> are used as the target
keyspace/table name. So for instance, to load an sstable named
Standard1-g-1-Data.db into Keyspace1/Standard1, you will need to have the
files Standard1-g-1-Data.db and Standard1-g-1-Index.db into a directory
/path/to/Keyspace1/Standard1/.
-alg,--ssl-alg Client SSL: algorithm
(default: SunX509)
-ap,--auth-provider custom AuthProvider
class name for cassandra authentication
-ciphers,--ssl-ciphers Client SSL:
comma-separated list of encryption suites to use
-cph,--connections-per-host number of concurrent
connections-per-host.
-d,--nodes Required. try to
connect to these hosts (comma separated) initially for ring information
-f,--conf-path cassandra.yaml file
path for streaming throughput and client/server SSL.
-h,--help display this help
message
-i,--ignore don't stream to this
(comma separated) list of nodes
-idct,--inter-dc-throttle inter-datacenter
throttle speed in Mbits (default unlimited)
-ks,--keystore Client SSL: full path
to keystore
-kspw,--keystore-password Client SSL: password
of the keystore
--no-progress don't display
progress
-p,--port port used for native
connection (default 9042)
-prtcl,--ssl-protocol Client SSL:
connections protocol to use (default: TLS)
-pw,--password password for
cassandra authentication
-sp,--storage-port port used for
internode communication (default 7000)
-ssp,--ssl-storage-port port used for TLS
internode communication (default 7001)
-st,--store-type Client SSL: type of
store
-t,--throttle throttle speed in
Mbits (default unlimited)
-ts,--truststore Client SSL: full path
to truststore
-tspw,--truststore-password Client SSL: password
of the truststore
-u,--username username for
cassandra authentication
-v,--verbose verbose output

You can provide cassandra.yaml file with -f command line option to set up
streaming throughput, client and server encryption options. Only
stream_throughput_outbound_megabits_per_sec, server_encryption_options and
client_encryption_options are read from yaml. You can override options
read from cassandra.yaml with corresponding command line options.

[EmViKay]

Also if i change target hosts from cassandra1 to IP i have this one:
WARNING:root:Deleting directory (/tmp/restore_cassandra/test3)...
Found 68 files, with total size of 19.6KB.
WARN 16:56:30,738 Small commitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 3758. You can override this in cassandra.yaml
WARN 16:56:31,298 Small cdc volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in_mb to 1879. You can override this in cassandra.yaml
WARN 16:56:33,411 Only 10.885GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
Established connection to initial hosts
Opening sstables and calculating sections to stream

Summary statistics:
Connections per host : 1
Total files transferred : 0
Total bytes transferred : 0.000KiB
Total duration : 28492 ms
Average transfer rate : 0.000KiB/s
Peak transfer rate : 0.000KiB/s

[gamunu]

First issue the cassandra1 hostname not resolvable. See: Unknown host: cassandra1: Name or service not known

cassandra_snap is using nodetool for restore operations. looks like this is generated from the nodetool becuase of low disk space. Try clearing old snapshots and use nodetool flush command to write memory pages to disk.

[EmViKay]

hmm. Maybe i didn't understand.
Could you please tell me where i should start Restore your data sstableloader (automatic) proccess?
In node where i need restore? or in no cassandra host?

[EmViKay]

I've started this command on node, where i need to restore.
cassandra-snapshotter --s3-bucket-name=xxx --s3-bucket-region=us-east-1 --s3-base-path=mycluster restore --target-hosts=192.168.15.46 --keyspace test3 --snapshot-name 20180501121212

After that command downloaded files into /tmp/restore_cassandra/test3. And didn't move those files to cassandra dir, Can you tell me why and how I can find what i did wrong?

There are all info from console:
cassandra-snapshotter --s3-bucket-name=xxx --s3-bucket-region=us-east-1 --s3-base-path=mycluster restore --target-hosts=192.168.15.46 --keyspace test3 --snapshot-name 20180501121212
WARNING:root:Deleting directory (/tmp/restore_cassandra/test3)...
Found 68 files, with total size of 19.6KB.
Established connection to initial hosts
Opening sstables and calculating sections to stream

Summary statistics:
Connections per host : 1
Total files transferred : 0
Total bytes transferred : 0.000KiB
Total duration : 4438 ms
Average transfer rate : 0.000KiB/s
Peak transfer rate : 0.000KiB/s