This release notes file describes new and important changes to the frontend services, toolbox, backend services and the knowledge graph of Thoth.
Release cycles may vary from 2 to 4 weeks, depending on complexity of features or capacity of the team! For the current state of planning, have a look at our Projects.
For all the nitty gritty details of our features (backend or services) have a look at our Sprint Demo recordings.
With this "Cumulative update Release" we announce the continuous update of all Thoth Services running on Operate First.
For the past six weeks, we have rolled out a release every two weeks! Most of the updates were ongoing service maintenance and service improvements. With this release we have added some new features.
See Thoth Application Kustomize manifests for a definitive list of all the versions used in this release.
The Prescriptions project keeps a database of known issues in Python open-source projects. The database is used in Thoth to resolve high quality Python software stacks. Have a look at our introduction video on YouTube.
If you would like to write a prescription for resolver, check the following docs.
Besides all the continuously applied dependency updates (Thanks Kebechet!!) we have added the capability to manage
virtual environments and added a thamos install command,
this is accompanied by the earlier added add command.
With this release, we are happy to announce that our service has moved to a publicly accessible location! We have invested a lot of work to migrate all our services to an OpenShift platform that is maintained by our friends of Operate First: Bots and Cyborgs, Pipelines and CI, Prow and all our GitHub applications have been migrated to run on the MOC! Thanks to all our supporters in these two great projects!
See Thoth Application Kustomize manifests for a definitive list of all the versions used in this release.
We added another tool to aggregate more information and learn from them: a bot that watches for builds done in an OpenShift cluster and automatically submits container images and build logs to Thoth Service. This bot helps Thoth to aggregate new knowledge about build failures and possible package issues.
This is something that could even be deployed on you on-premise cluster and help with extending the Thoth knowledge graph.
As we can not open up the chat in general, we would like to ask you to open an issue on GitHub, and we will invite you to our channel. Please include the Google ID you would like us to send the invite to.
As always, we have released new container images via our OpenShift Pipelines based release toolchain, please find all the current container images on our organization on quay.io.
Since version 0.21 the Thoth S2I builder images are label to include security-related knowledge into account while
generating a software stack recommendation (or advise as we call it). As a result of a large knowledge generation
activity during December and January, we are now able to provide security (specifically CVE and bandit based)
information advise! The effect of this large gain in knowledge will be more secure software stacks! You can get
these recommendations by selecting the appropriate recommendation_type (latest or security) via thamos'
configuration, see thamos documentation on recommendation types
for more details.
- Remove use-before-declared linter warning (#390)
- Buildlog analysis trigger (#393)
- Manual update of dependencies (#405)
- analsysis->analysis (#410)
- remove message contents (link instead) (#409)
- decs are applied inside-out (#424)
- Add metric schema (#428)
- Standardize metrics for revision check (#434)
- Do not use mutable arguments in functions (#391)
- removed bissenbay, thanks for your contributions!
- Update dependencies to have more recent thoth-common (#418)
- retry on exceptions and other error handling (#389)
- State python_requires in the compatibility section of docs (#1619)
- Implement a sieve that filters out TensorFlow==2.4.0 on non-AVX2 CPU (#1617)
- Add justification to stack info if a package with CVE is avoided (#1611)
- Recommend TensorFlow 2.4 based on CUDA support (#1605)
- Relock so that typing extensions have the right environment marker (#1607)
- Fix testsuite for Python 3.8 (#1603)
- Introduce THOTH_CONFIG_CHECK environment variable (#1592)
- Update TensorFlow symbols database (#1587)
- port to python 38
- Add specific labels to issues (#286)
- Fix/dependency update author (#290)
- PullRequestDiscussion entity (#302)
- add dominik to approvers (#305)
- Add explicitly thoth-pytest38 (#303)
- Multiple entities now passed as comma sep. string (#296)
- Check if readme exists (#319)
- Check that patch can be None (#320)
- Add title and body attributes for inspection (#298)
- Add requirements for PyPI, remove unnecessary dependencies (#310)
- Adjustments to use new build analysis endpoint (#183)
- port to python 38 (#176)
- ⛄ support pre-commit (#174)
- Remove latest versions limitation (#171)
- added a 'tekton trigger tag_release pipeline issue'
- Catch the manifest exception as warning
- Fixing traceback raise issue with warning
- Remove redundant if statement
- Fixing @ issue in image push to quay
- Flexibility to push images to quay
- Happy new year!
- Use RHEL instead of UBI
- Update Thoth configuration file and Thoth's s2i configuration
- updated templates with annotations and param thoth-advise-value
- Propagate deployment name for sentry environment
- openshift deployment templates changed
- Distinguish TLS verification flag
update few fixes and also prometheus metrics set operation- 🎶 build-watcher is updated to send prometheus metrics
- 🎉 update to watch the entire build(images, base_image & buildlog)
- Update zuul pipeline to use the new version trigger build job
- Added a required field for deployment of dc and imagestream in different namespace
- Use versions of libraries from PyPI
- Report environment type of images submitted for analysis
- Propagate environment type on provisioning
- Create service account in deployment
- Provide Ansible playbooks
- State service account configuration in README
- State presence of s2i container
- Implement process pool of workers
- Add skopeo binary
- Add ability to push containers to an external registry
- Do not share OpenShift instance across namespaces
- Introduce event producer following workqueue pattern
- Do not clash with env var used by thoth-common
- State Thamos env var to disable TLS warnings
- Do not propagate credentials if user did not request analysis (#219)
- Extend README file with links (#218)
- No submit parameters (#217)
- Adjust deployment templates and README
- Provide parameters to avoid submitting specific inputs (#215)
- Fix typing in the application (#212)
- Use s2i-thoth-ubi-8-py38 as a base image (#195)
- Fix logged entries which might be None (#202)
- Add pull-request template (#196)
- Tweak environment variables supplied (#203)
- Fix pre-commit issues and bump black version (#197)
- Fix API version in sources (#199)
- Relock to fix typing extensions issue caused by Pipenv resolver (#191)
- 🐛 fixed the webhook url of the build trigger job
- Push to registry only if the push registry was provided (#204)
- Improve error message reported to the user (#1588)
- Provide version identifier for Kebechet and expose it in logs (#190)
- sa and rolebinding creation separated
- Do not run adviser from bc in debug mode
- Distinguish between runtime and buildtime images monitored
- Fixes and improvements during dh-jupyterhub deployment
- Fixes needed for correct pushing
- Add missing parameter, make TLS disabling of warnings parametrizable
- Minor improvements in template
None, please file an issue if you hit any, our join our Thoth Station Developer Chat
After releasing some features to our backend and knowledge graph last time, we are now happy to announce the general availability of a new GitHub App called "Khebhut"! Please feel free to join our newly created developer chat.
After a rework on our backend, we are happy to announce that GitHub has approved our newly updated Junior Cyborg Developer application "Khebhut"! The main activity of Khebhut is to keep your dependencies up to date, create releases of your software (incl. bouncing version numbers). Please add Khebhut to your repository and give it a try, let us know what you think!
We have created a public channel on Google Hangouts: Thoth Station Developer Chat, this is meant as an open and interactive channel to ask questions with regards to Thoth's usage, development, deployment. As we are a very Python focused community, feel free to ask how we (the cyborgs and the humans) can help!
We have updated thamos with two new commands:
check- to check.thoth.yamlandPipfilefor consistency: for example, is the same Python version used in both files.- overlay directories - so that we could handle multiple different software stacks per repository
The toolbox container image has been updated to v0.5.10, containing the latest versions of micropipenv, thamos and thoth-glyph.
Try it out or your Fedora or Red Hat Enterprise Linux Workstation:
[user@hostname ~]$ toolbox create --image quay.io/thoth-station/thoth-toolbox:v0.5.10
Created container: thoth-toolbox
Enter with: toolbox enter --container thoth-toolbox-v0.5.10
[user@hostname ~]$This will create a container called thoth-toolbox-v0.5.10.
[user@hostname ~]$ toolbox enter --container thoth-toolbox-v0.5.10
⬢[user@toolbox ~]$ thamos version
Thamos Client version: 1.5.0
...
...- Add a generic alias pseudonym unit
- Add a link to Jupyter Notebook demonstrating pipelines
- Propagate statistics to the final report
- Handle SIGUSR1 handler to stop exploitation phase (#1527)
- Introduce a sieve for filtering out incompatible TensorFlow for Py3.9 (#1528)
- Provide stack info in security indicators
- Increase verbosity to see where inspections are triggered
- Add links to termial random
- Implement a sieve that filters out TensorFlow releases based on API (#1560)
- Consider library usage for TF 42475 wrap (#1564)
- Add a pipeline unit wrap for slow keras embedding layer (#1558)
- Add missing link to user-stack scoring justification (#1556)
- Include dependency if at least one lib always requires it (#1594)
- properly JSON formatted advised manifest changes (#1584)
- Add a warning to TF API (#1581)
- h5py==3 causes troubles also on TensorFlow 2.3.1 (#1576)
- Link Jupyter notebook showing TD-learning and MCTS predictors
- A pipeline unit that suggests not to use h5py>=3 with TF==2.1 (#1529)
- Add links to TDS and Jupyter Notebook
- Implement a boot pipeline unit for checking Pipfile hash (#1571)
- Report warning if Python versions do not match (#1565)
- Adjust tests accordingly
- ✨ remove the Zuul config file, as we dont use Zuul anymore
- Adjust tests for stack_info provided by security indicators
- Implement a boot pipeline unit for checking Pipfile hash (#1571)
- Report warning if Python versions do not match (#1565)
- Improve message logged when reporting resolver's progress (#1569)
- Match score of the user's stack printed with the final score reported (#1570)
- Add a wrap that notifies about a bug when mutliple instances of TF are running (#1559)
- Handle exception raised when the given record was not found
- Fix issue when signal is sent in one call in livenenss.py
- Handle cannot produce stack exception so results are not overwritten
- Fix 404 for queued requests (#1124)
- Place dependencies under package to conform other endpoints (#1120)
- Place metadata info into Python packages section (#1119)
- Expose all packages (#1104)
- Messaging 0.7.13 (#1136)
- Removed list endpoints (#1143)
- Make sure the advise endpoint is available when becoming ready (#1134)
None, please file an issue if you hit any, our join our Thoth Station Developer Chat!
With this release we have focused on knowledge generation: a) by better connecting the backend components and b) by broadening the security related indicators.
All backend features feed directly into the quality of Thoth's services, more specifically Kebechet and Thamos.
See Thoth Application Kustomize manifests for a definitive list of all the versions used in this release.
None, please file an issue if you hit any!
All our end-user tools (like thamos or glyph) are available as part of the Thoth Toolbox container image.
Thoth's investigator is a Kafka based component that consumes all messages produced by Thoth components and reacts to them by scheduling Argo Workflows.
It has a monitoring system in places that allow Thoth's DevOps team to observe what is happening within Thoth in terms of Kafka, OpenShift, and Argo for all the different components.
Depending on the type of message received by Investigator, a workflow is scheduled to increase or update the knowledge stored by Thoth. As always, the readme is a great source of detailed information.
thoth-glyph for this release:
0 correct link (#345) features
1 confluent rework (#344)* use wip: messaging ch... perfective
2 add kebechet run url (#342)* added kebechet ru... features
3 :pushpin: automatic update of dependency hypot... features
4 :pushpin: automatic update of dependency thoth... features
5 :pushpin: automatic update of dependency thoth... features
6 add docs for thoth investigator (#330)* add do... features
7 :pushpin: automatic update of dependency hypot... features
8 :pushpin: automatic update of dependency thoth... features
9 remove producer from investigator (#329)* remo... unknown
10 :pushpin: automatic update of dependency mypy ... features
11 :pushpin: automatic update of dependency thoth... featuresWe have introduced Security Indicators: all package releases
observed by Thoth are augmented with bandit related information. These information
is used during advise generation to add or remove packages from candidate stacks. Security Indicators are taken into
consideration when using the 'security' recommendation type
with thamos advise.
Adviser is Thoth's recommendation system, depending on the recommendation type, it takes a set of observations into account when resolving a software stack and generating a recommendation to the user.
This release features the technical requirements to include Security Indicators into a software stack resolution.
If you would like to interact with Thoth from a user's perspective, check Thamos.
Adding thoth-glyph to our toolbox container image enables a developer to get a quick view of what changes in which categories. The Glyph readme is a great source of information, please go ahead and pull the container image from https://quay.io/repository/thoth-station/thoth-toolbox?tab=info
We have done a lot of 'internal' updates and maintenance, focusing on renewing the way Thoth handles learning about new releases of packages.
See Thoth Application Kustomize manifests for a definitive list of all the versions used in this release.
Thoth Investigator is an agent sent out by Thoth to seek new information on packages, that will yield observations and knowledge to Thoth.
It is called by Thoth components to gather new messages after investigations about possible observations on packages.
Thoth Investigator centre of investigation receives those messages and after further investigation decides what actions need to be taken depending on the messages received, so that Thoth can increase its knowledge.
This release of investigator is migrating more components of Thoth to a Kafka-based messaging. In addition to that, we re-schedule adviser's Argo Workflows that have not created results before. The re-scheduled workflow will create more meaningful advise as Thoth might have learned about packages being used (which might have been unknown on the first run) and help Thoth keeping its knowledge up to date.
Thoth's Service Level Objectiv (SLO) Reporter purpose is to share Thoth's achievements and behaviour with the outside world.
We have extended it's README and developers guide, so that understand it's concepts and components get easier. A lot of Thoth internal observability has been added to a large set of components, all metrics are surfacing via SLO reporter into Grafana dashboards.
None are known, please file issues if you hit any!
These enhancements have been deployed and are available to Qeb-Hwt immediately.
First of all, why do we use CalVer for the core repository? Because the Thoth Services and Tools consist of a large set of GitHub-hosted repositories, Python modules, container images and even RPMs. Any of these components might yield to a release at any time. This release note file covers the aggregation of important releases in the contexts of
- Thoth Service - our back end services, the knowledge generation
- the toolbox - command line tools and their packages
Starting with this release, we publish a human-readable list of justifications. This list is referenced from any thamos advise output on the terminal, or any pipeline integration. Its purpose is
to show the knowledge contained in Thoth's services and to act as a jump-page to more detailed and background information.
Especially the jump-page character is important to users, as we don't want to clutter the terminal/pipeline output, but
want to reveal to results of our reinforcement learning process in a transparent way.
None are known, please file issues if you hit any!
Adviser is Thoth's recommendation engine and software stack generator.
- a review of CPU types, and the way we create recommendation for AVX2 and Tensorflow has been done
- some new TensorFlow related observations/justfications have been created
- a recommendation to configure MKL correctly was added
- general stabilization of Argo-Workflows based backend services
- adviser now supports advises specific for manifest changes (e.g. adjusting environment variables of OpenShift DeploymentConfigs)
These enhancements have been deployed and are available to Qeb-Hwt immediately.