-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Capitalize in the beggining of a sentence * Capitalize document filenames * Add missing periods * Add more markdown formatting where appropriate. * Add links for URLs and references to documents in the repo * Remove unnecessary indentation * Wrap at 80 characters No changes in the instructions or wording were done is this commit.
- Loading branch information
Tiago Gomes
committed
Oct 12, 2016
1 parent
d8aba23
commit 4b02337
Showing
6 changed files
with
262 additions
and
288 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,236 @@ | ||
| GitLab: [](https://gitlab.com/baserock/ybd/commits/master) | ||
|
|
||
| Github: [](https://travis-ci.org/devcurmudgeon/ybd) | ||
|
|
||
| # ybd | ||
|
|
||
| ybd is a tool for building integrated software stacks. It does four things: | ||
|
|
||
| - parse yaml files which describe integrated collections of software components | ||
| - gather source code (from git repos) for a user-specified target collection | ||
| - build the collection | ||
| - optionally deploy the collection (if the yaml files describe where/how to | ||
| deploy) | ||
|
|
||
| Currently ybd understands the semantics of yaml | ||
| [definitions](http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/) | ||
| from the [Baserock](http://wiki.baserock.org) project. | ||
|
|
||
| The total codebase for ybd is only ~ 2200 lines of Python. even so ybd can | ||
| reliably build all systems in Baserock's definitions.git, i.e. all of the FOSS | ||
| components required for Linux appliances up to and including: | ||
|
|
||
| - minimal Linux systems | ||
| - self-hosting development systems | ||
| - GENIVI Baseline and GENIVI Demo Platform systems for x86 and ARMv7 | ||
| - OpenStack appliances | ||
| - OpenWRT appliances | ||
|
|
||
| There are Concourse pipelines using ybd to integrate various examples at | ||
| <concourse.baserock.org> ... | ||
|
|
||
| [<img src="http://i.imgur.com/N193G9C.png" height="400" width="600">](http://concourse.baserock.org) | ||
|
|
||
| ybd is a simple start-point for building, deploying, learning and experimenting | ||
| with definitions, algorithms and functionality on Baserock projects. With a | ||
| little work it can be used to build other software stacks too. | ||
|
|
||
| ybd is under development. things will change :) If you're trying ybd for the | ||
| first time please start with the latest tag, not master. | ||
|
|
||
| ybd is designed for Linux, but can be used on other environments (eg MacOS) | ||
| using Vagrant and VirtualBox. | ||
|
|
||
| ## Quick Start | ||
|
|
||
| On Fedora, Ubuntu, Debian (as root... hoping to fix this for non-root soon): | ||
|
|
||
| git clone git://gitlab.com/baserock/ybd && cd ybd | ||
| # checkout latest tag | ||
| git checkout `git describe --tags $(git rev-list --tags --max-count=1)` | ||
| sudo sh ./install_dependencies.sh | ||
| cd .. && git clone git://git.baserock.org/baserock/baserock/definitions | ||
| cd definitions | ||
|
|
||
| Or using Vagrant with VirtualBox: | ||
|
|
||
| git clone git://gitlab.com/baserock/ybd && cd ybd | ||
| # checkout latest tag | ||
| git checkout `git describe --tags $(git rev-list --tags --max-count=1)` | ||
| vagrant up | ||
| vagrant ssh | ||
| sudo -i | ||
| cd /src/definitions | ||
|
|
||
| Once there you can run (as root) | ||
|
|
||
| ../ybd/ybd.py $1 $2 | ||
|
|
||
| where | ||
|
|
||
| - `$1` is relative path to a chunk, stratum, system or cluster definition file | ||
| - `$2` specifies the architecture you're building/deploying for. | ||
|
|
||
| Note that all of the current baserock definitions are for native builds, no | ||
| cross-compile. So if you're on a PC/Mac it may be that your only option for | ||
| `$2` is `x86_64`. | ||
|
|
||
| Some examples to try: | ||
|
|
||
| # on an x86_64 laptop running linux, to build a build-system... | ||
| ../ybd/ybd.py systems/build-system-x86_64.morph x86_64 | ||
|
|
||
| # on the same laptop, to build gcc (and its dependencies) | ||
| ../ybd/ybd.py gcc x86_64 | ||
|
|
||
| # on the same laptop, to build the genivi stratum (and its dependencies) | ||
| ../ybd/ybd.py strata/genivi.morph x86_64 | ||
|
|
||
| # on a jetson, to build a GENIVI baseline system... | ||
| ../ybd/ybd.py systems/genivi-baseline-system-armv7lhf-jetson.morph armv7lhf | ||
|
|
||
| # on anything, to build (and deploy?) parts of baserock release... | ||
| ../ybd/ybd.py clusters/release.morph x86_64 | ||
|
|
||
| # in a baserock devel vm (x86_64), to build and deploy a self-upgrade... | ||
| ../ybd/ybd.py clusters/upgrade-devel.morph x86_64 | ||
|
|
||
| ybd generates a lot of log output to stdout, which hopefully helps to explain | ||
| what is happening. If you need a permanent log then try | ||
|
|
||
| ../ybd/ybd.py clusters/upgrade-devel.morph x86_64 | tee some-memorable-name.log | ||
|
|
||
| More information about ybd's dependencies is in | ||
| [dependencies](docs/Dependencies.md). | ||
|
|
||
|
|
||
| ## Configuration | ||
|
|
||
| ybd is designed to be run from the command line and/or as part of an automated | ||
| pipeline. all configuration is taken from conf files and/or environment | ||
| variables, in the following order of precedence: | ||
|
|
||
| YBD_* environment variables # if found | ||
| ./ybd.conf # if found | ||
| $path/to/ybd.py/ybd.conf # if found | ||
| $path/to/ybd.py/ybd/config/ybd.conf # default, as provided in the ybd repo | ||
|
|
||
| This means you can set custom config via env vars, or the definitions top-level | ||
| directory, or the ybd top-level directory, without having to modify the | ||
| supplied default [ybd.conf](ybd/config/ybd.conf). | ||
|
|
||
| NOTE we recommend you create your own ybd.conf file. that way you can git merge | ||
| new versions of ybd + definitions, and your custom settings will always take | ||
| precedence, with no possibility of a merge conflict. | ||
|
|
||
| To set config via environment variables, each must be prefixed with `YBD_` | ||
| and using `_` instead of `-`. ybd will strip the `YBD_` prefix and | ||
| convert `_` to `-`. For example | ||
|
|
||
| export YBD_artifact_version=1 # artifact-version: 1 | ||
| export YBD_log_verbose=True # log-verbose: True | ||
| export YBD_instances=2 # instances: 2 | ||
|
|
||
| ybd does not support unix-style --flags so far. If enough people complain about | ||
| that, it can be fixed. | ||
|
|
||
| For details about the config options themselves, see | ||
| [ybd.conf](ybd/config/ybd.conf). | ||
|
|
||
| ## Interesting Features | ||
|
|
||
| ### Run ybd in Parallel | ||
|
|
||
| ybd can fork several instances of itself to parallelise its work. There is no | ||
| intelligence in the scheduling - all of the forks just randomise their | ||
| build-order and try to build everything. For building a set of overlapping | ||
| systems in parallel on a many core machine this proves to be quite effective. | ||
| For example on a 36-core AWS c4.8xlarge machine, 4 racing instances of ybd can | ||
| build all of the x86_64 systems in definitions/clusters/ci.morph much faster | ||
| than a single instance. | ||
|
|
||
| To try it, just set the `instances` config variable, for example | ||
|
|
||
| # as an environment variable... | ||
| export YBD_instances=4 | ||
|
|
||
| # or in a ybd.conf file... | ||
| instances: 8 | ||
|
|
||
| You should probably think about setting `max-jobs` too, taking into account | ||
| your workloads and host machine(s). If `max-jobs` is not set, ybd will default | ||
| it to number-of-cores/instances. | ||
|
|
||
| ### kbas Cache Server | ||
|
|
||
| kbas is a basic server which can be used to allow other users to access | ||
| pre-built artifacts from previous or current runs of ybd. See kbas/kbas.py for | ||
| the code. With minimal configuration it can serve artifacts to instances of ybd | ||
| on other machines, and also receive uploaded artifacts. | ||
|
|
||
| To launch kbas, just do | ||
|
|
||
| cd kbas && ./kbas.py | ||
|
|
||
| By default ybd is configured to check for artifacts from a kbas server at | ||
| <http://artifacts1.baserock.org:8000/>. | ||
|
|
||
| Config for kbas follows the same approach as ybd, defaulting to config in | ||
| `kbas/config/kbas.conf`. | ||
|
|
||
| NOTE: the default password is 'insecure' and the uploading is disabled unless | ||
| you change it. | ||
|
|
||
| ### Concourse Pipelines | ||
|
|
||
| [WORK IN PROGRESS] ybd can generate concourse pipelines - see the code at | ||
| ybd/concourse.py | ||
|
|
||
| To start a local concourse instance: | ||
|
|
||
| # in your definitions directory | ||
| vagrant init concourse/lite | ||
| vagrant up | ||
| # generate pipeline, run concourse.py (same arguments as ybd.py) | ||
| python ../ybd/ybd/concourse.py <target> <arch> | ||
| fly -t local login -c http://192.168.100.4:8080 | ||
| fly -t local set-pipeline -p <target> -c <target>.yml | ||
|
|
||
| You can view the local pipelines at <http://192.168.100.4:8080>. | ||
|
|
||
| ## TODO | ||
|
|
||
| In no particular order, here's a list of things that may be interesting to try | ||
| doing to/with ybd: | ||
| - pip install | ||
| - Deterministic (bit-for-bit) build | ||
| - Fix deployment: ybd currently can deploy systems, subsystems (test this?) and | ||
| upgrades, but it's a mess: | ||
| - can't select which subsystems to deploy (need to iron out the definitions) | ||
| - alter the deployment parameters (templating, or commandline?) | ||
| - deploy other-architecture stuff | ||
| - Run without root privileges on non-Linux environments (eg Mac OS - needs a | ||
| different isolation method instead of linux-user-chroot). | ||
| - Test assumptions by creating definitions for other OS software sets, eg | ||
| Aboriginal Linux and non-Linux systems. | ||
| - Test/fix to work on old versions of definitions, and on morphs.git (and | ||
| separate out the idiosyncrasies, so code is tidy for future changes) | ||
|
|
||
| ## Project Guidelines | ||
|
|
||
| - Contributions are extremely welcome - feedback, ideas, suggestions, bugs, | ||
| documentation will be appreciated just as much as code. | ||
| - Key criteria for code contributions are: | ||
| - 'Code LESS: every line creates a work-chain. Code is a liability, not an | ||
| asset' | ||
| - think hard before adding dependencies | ||
| - code should be tested and pass pep8 | ||
|
|
||
| ## License | ||
|
|
||
| - License is GPLv2 but other licensing can be considered on request. | ||
| - Most of the copyright is currently Codethink but don't let that put you off. | ||
| There's no intent to keep this as a Codethink-only project, nor will there be | ||
| any attempt to get folks to sign a contributor agreement. contributors | ||
| retain their own copyright. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.