Are Logs from the Pods and Events also stored?

[manuelwallrapp]

Hi

We plan to implement something similar, where we store PipelineRuns, TasksRuns, Logs and Events from the Taskrun Pod in a KeyValue Store. My question is, do you also store Logs and Events from the TaskRun Pod?

Best regards.. Manu

[adambkaplan]

Logs is a new feature that recently just merged, and we hope to make available in v0.5.0 when it is released. See TEP-0117 for some of the design behind it.

Events is not something that we currently store, but I can see this being relevant for capturing the history of a PipelineRun/TaskRun. @dibyom @alan-ghelardi perhaps we open up a feature issue for discussion?

[dibyom]

Yeah storing events could be useful

[manuelwallrapp]

For us it would be essential to have logs and events. We built an opinionated Tekton Build System with frontend and backend and using it already in bigger scale - we also plan to open source it.

You can see a very short introduction at the last RedHat devnation GitOps Talk from us:
https://www.youtube.com/live/SpRSndQCl2I?feature=share&t=5690

Since we have a lot of builds and using the PipelineRuns as a kind of Metadata Database to display old builds but deleting the Pods and PVC underneath, we could use tekton-results instead. Our Build cluster got slowly a little overwhelmed by the amount of resources.

The logs and events we already stream and store locally in a PVC, now we thought to do the same with PipelineRuns. Then we considered to use tekton-results. We also thought about using other solutions as a Persistent Cache instead of writing the files directly to the filesystem. Somewhere where we can stream data and listen to cache changes.
So for us it is kind of strategical to either use something like tekton-results or build our own archive solution. I think if we had logs, events, PipelineRuns and TaskRuns all together over a API which we can query, that would meet our needs.

So Events are also essential for us to store and query.

[alan-ghelardi]

+1 for opening a feature request to store events. There might be a few challenges to be aware though to deal with events at scale. Recently we ran into some issues due to the insane number of events our CI/CD cluster was generating. In larger clusters, the Watcher may have issues to sync informers upon startup and it may cause a delay to store PipelineRuns and TaskRuns. So, definitely, it's a topic which deserves a deeper discussion in terms of design.

[alan-ghelardi]

@manuelwallrapp, we're using Tekton Results at scale at Nubank as well. We're reaching to 1 million of TaskRuns a week and Results has been behaving well with this volume. We use it for storing PipelineRuns, TaskRuns and other resources that make up our CI/CD ecosystem as well as for keeping the number of objects in the cluster under control. The latest code on the main branch (still officially unreleased) has a bunch of fixes to allow that.

We're building a custom API on top of Results as well and I've been working to improve the search capabilities of Results mostly to work at scale. So, if you people decide to adopt it, it'd be a pleasure hearing your feedback and collaborating to make the project better.

[manuelwallrapp]

@alan-ghelardi thank you very much for your feedback. We also generate quite some load with Tekton. The problem with the informers we know and had/have. But we built our own Garbage Collector and store Events and Logs on a PVC. But it is true that the Watcher causes us also timeouts if we have too many PipelineRuns lying around. So if you use tekton-results in large scale, it is defenitely something for us. For the moment we want to solve the backup of PipelineRuns and TaskRuns so we can Garbage Collect them after a day and it seems tekton-results does the job. I am now setting it up and integrating it into our build software. We are building the Tekton build system for the Swiss Federal Railways (SBB).

Of course it would be great to have Events and Logs managed by tekton-results, so we can dump our own PVC solution.

[adambkaplan]

Closing this discussion:

• Logs were introduced in v0.5.0 (and have important fixes coming in v0.6.0) 🎉
• Events was added as a feature request see Record Kubernetes Events related to PipelineRuns/TaskRuns #439 ✔️