Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to deploy velero #1452

Closed
gkdba87 opened this issue May 6, 2019 · 28 comments
Closed

Unable to deploy velero #1452

gkdba87 opened this issue May 6, 2019 · 28 comments

Comments

@gkdba87
Copy link

gkdba87 commented May 6, 2019

HI
I am trying to install Velero on kubernates using below url
(https://github.com/heptio/velero/blob/master/docs/ibm-config.md)

command: velero install --bucket team-cicd --provider aws --secret-file ./credentials-velero --backup-location-config s3ForcePathStyle="true",s3Url=http://laas.com --namespace=dev.
following are the deploy and pod logs
deploy logs:
image

pod logs :
image

Please look into this issue, do the needful.

@typhoonrr
Copy link

Hey, i have exactly the same error.

An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = unable to get all required environment variables: the following keys do not have values: AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_SUBSCRIPTION_ID; error getting backup store for location "velero": rpc error: code = Unknown desc = config has invalid keys [region]; valid keys are [resourceGroup storageAccount bucket]

Anyone has an idea how to edit those values?

Marc

@skriss
Copy link
Member

skriss commented May 6, 2019

@gkdba87 can you provide the output of velero backup-location get -o yaml?

@skriss
Copy link
Member

skriss commented May 6, 2019

@typhoonrr it sounds like you're missing a bunch of the necessary credentials and config for Azure. If you're using the v1.0 beta, did you follow the instructions here? https://heptio.github.io/velero/v1.0.0-beta.1/azure-config

@skriss skriss added this to the v1.0.0 milestone May 6, 2019
@typhoonrr
Copy link

@skriss yeah i had an issue with my secret, it's working now.

@gkdba87
Copy link
Author

gkdba87 commented May 7, 2019

image
@skriss
i could try to install Velero from scratch. please do needful

@gkdba87
Copy link
Author

gkdba87 commented May 7, 2019

@skriss
Its pointed to wrong s3URL, can i change ?

@gkdba87
Copy link
Author

gkdba87 commented May 7, 2019

@skriss
please find updated state
image

@skriss
Copy link
Member

skriss commented May 7, 2019

@gkdba87 I'm a little confused now. Are you trying to install v0.11.0 or v1.0.0-beta.1? Also, in your latest screenshot it looks like the cloud-credentials secret does not exist in the velero namespace or doesn't have the right content. Are you still using velero install, or something else?

@gkdba87
Copy link
Author

gkdba87 commented May 9, 2019

@skriss Now i have successfully installed Velero v1.0.0-beta.1.
image

  • Above setup was required empty bucket

@gkdba87
Copy link
Author

gkdba87 commented May 9, 2019

@skriss Now i am trying to to create backup of namespace , using following command

velero backup create bkp1 --include-namespaces demo1

But aim geeting errors. Can you please look into this and give suggestion.
image

image

@skriss
Copy link
Member

skriss commented May 9, 2019

@gkdba87 we don't have a VolumeSnapshotter for IBM - if you want to do volume backups, you'll need to use restic. You should kubectl -n velero delete volumesnapshotlocations --all to remove any configs for the VolumeSnapshotters.

@gkdba87
Copy link
Author

gkdba87 commented May 9, 2019

@skriss Thanks for the update. Can i install With restic( Re-install velero and Install velero with Restic)
please correct me if i am wrong

@skriss
Copy link
Member

skriss commented May 9, 2019

yeah, you can reinstall and use the --use-restic flag. Note that you'll still need to kubectl -n velero delete volumesnapshotlocations --all afterwards - we don't yet have a way to disable those from being created. It's being worked on in #1462

@gkdba87
Copy link
Author

gkdba87 commented May 9, 2019

root@cd:/home/venu> kubectl delete volumesnapshotlocations --all -n cicd
No resources found

@skriss
Copy link
Member

skriss commented May 9, 2019

is that the namespace you have velero deployed into? you can also try kubectl delete volumesnapshotlocations --all --all-namespaces -- but as long as there are none, you're good.

@gkdba87
Copy link
Author

gkdba87 commented May 13, 2019

@skriss i have deployed velero + restic in one of our namespace. Some of pods are going to error state. and also Damon-set also not available state. I have limited CPUs and Memory for Namespace. How i can restrict Resources @ "velero install" command level . Can plese look into following error(**velero Pod is up and running fine)

image

@skriss
Copy link
Member

skriss commented May 13, 2019

Can you kubectl describe the pods that are crashing?

Right now we don't have support for adding requests/limits via velero install. You can always dump the generated YAML via velero install ... --dry-run -o yaml and manually add what you need, then kubectl apply it.

cc @nrb -- do we want to add request/limit support?

@gkdba87
Copy link
Author

gkdba87 commented May 13, 2019

@skriss un-healthy describe pod
image
healthy describe pod output
image

@skriss
Copy link
Member

skriss commented May 13, 2019

I don't see anything showing What the problem is -- what about kubectl logs?

@gkdba87
Copy link
Author

gkdba87 commented May 13, 2019

unhealthy pod logs
image
Heathy pod logs
image

Now i get yaml file using dry-run, and i will set cpu limits and resource. see wat happend?

@nrb
Copy link
Contributor

nrb commented May 13, 2019

@skriss I’m not sure on adding those flags to the install command. Wouldn’t be hard to do, but the more we add the more we’re maintaining in parallel to the Helm chart

@skriss skriss modified the milestones: v1.0.0, v1.x May 20, 2019
@skriss
Copy link
Member

skriss commented Jun 6, 2019

xref #94

@ghost
Copy link

ghost commented Jun 6, 2019

I'm having a similar issue.

$ velero backup-location get
WARNING: error retrieving namespace from config file: stat /Users/vihanagarwal/.config/velero/config.json: permission denied
NAME      PROVIDER   BUCKET/PREFIX
default   aws        test-cluster-velero-backups

Here are the pod logs

An error occurred: some backup storage locations are invalid: backup store for location "default" is invalid: rpc error: code = Unknown desc = AccessDenied: Access Denied

@skriss
Copy link
Member

skriss commented Jun 6, 2019

@VihanAgarwal97 did you use velero install? I would double-check that the content of the file you provided for the --credentials-file flag is a valid AWS access key and that it's been granted IAM access to your object storage bucket

@ghost
Copy link

ghost commented Jun 6, 2019

Do you mean the --secret-file flag?

Here's my policy from IAM

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::test-cluster-velero-backups.s3.amazonaws.com/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::test-cluster-velero-backups.s3.amazonaws.com"
            ]
        }
    ]
}

Here is my credentials file (double checked for validity)

[default]
aws_access_key_id=<REDACTED>
aws_secret_access_key=<REDACTED>

Install command

	velero install \
    --provider aws \
    --bucket test-cluster-velero-backups \
    --secret-file ./velero-credentials \
    --backup-location-config region=${AWS_REGION} \
    --snapshot-location-config region=${AWS_REGION} \

@skriss
Copy link
Member

skriss commented Jun 6, 2019

Is test-cluster-velero-backups.s3.amazonaws.com valid syntax for specifying a bucket as a resource in IAM? I think you may want this to just be test-cluster-velero-backups.

@ghost
Copy link

ghost commented Jun 6, 2019

Ahh!! Thank you, that fixed it. The issue was with my terraform!

@skriss
Copy link
Member

skriss commented Jun 25, 2019

Closing as inactive; the feature request to add flags to velero install has been noted in #94.

@skriss skriss closed this as completed Jun 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants