Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

implement Gluon build-framework #765

Draft
wants to merge 82 commits into
base: main
Choose a base branch
from
Draft

implement Gluon build-framework #765

wants to merge 82 commits into from

Conversation

SvenRoederer
Copy link
Contributor

This PR is a basic implementation of the gluon build-framework for our build-process. This idea was mainly driven by looking for a way to solve Issue #696, as our current process provides no option to define board-specific packages. See targets/ath79-generic for an example to select ath10k drivers in place of upstream-selected ath10k-ct.

The framework also brings a feature to check that all required CONFIG_options are set and packages are enabled. This way I found some faults in OpenWrt-config we used (4c97589, a8bae20, 572abd8).

The code is yet not a full replacement of the existing build-scripts, but it demonstrates the principle and generates usable images. Looking forward to comments and improvements

@SvenRoederer
rename packages folder to imagetypes
1095594 
When we started to use some code of the gluon framework a minor conflict
of the useage of the packages-folder came up.
We use it to store the package lists for our images, Gluon uses this
folder to store the upstream repos. By this double-use there are always
the repos inseide the folder and causing git to report them as untracked
files. Adding the folder to .gitignore is also no option, as we need it
for our package lists.
So just lets move our files and adjust .gitignore.
@SvenRoederer
assemble_firmware.sh: adapt to renamed packages folder
01a0b58
@SvenRoederer
README: short description of this branch
2fe7a19
@SvenRoederer
import basic files from gluon (15. Feb. 2020)
2678494 
freifunk-gluon/gluon@f83d676
@SvenRoederer
gluon: (partial) add FLAG "GLUON_FOREIGN" to make the framework usabl…
be75b80 
…e for other types of firmwares

ignored changes in MAkefile

defining this FLAG (to "1") disables / skips some things

* including gluon-core packages
* strict inclusion of some packages
* per device sepecific kernel build
* a gluon specific target/generic_gluon will be considered
@SvenRoederer
adapt gluon: Makefile improve
652a0c4 
add targets:
* $(LUA):
* gluon-config:

define Vars:
* GLUON_SITEDIR
* GLUON_CONFIG_VARS
@SvenRoederer
Makefile: gluon import more definitions
5b49767 
* GLUON_TARGET
* GLUON_PACKAGES
* GLUON_TARGETSDIR
* OPENWRTMAKE
@SvenRoederer
scripts: fix permissions
5502a64
@SvenRoederer
Makefile: gluon import setup of LANG and LC_ALL
522c465
@SvenRoederer
MAkefile: [gluon] add target gluon-all which acts as all from origina…
14b8844 
…l gluon
@SvenRoederer
extend FOREIGN_BUILD to include different target/generic
bcd4e96
@SvenRoederer
Update upstream repos to openwrt-19.07.2 tag
81c3a42 
fixes: CVE-2020-8597, CVE-2013-1798, CVE-2019-14896, CVE-2019-14897

OpenWrt:
4668ae3bed OpenWrt v19.07.1: revert to branch defaults
a229907150 ramips: remove duplicate DEVICE_PACKAGES for TP-Link Archer C20i
aed6632d31 ramips: use tpt DTS trigger for TP-Link TL-MR3020 v3 and TL-WA801ND v5
3d1c84d424 ramips: reenable image creation for the D-Link DIR-645
eca8a2ee0d kernel: bump 4.14 to 4.14.169
2a844349fa kernel: add support for GD25D05 SPI NOR
d91b52b1a2 kernel: add missing symbol
887eb669f9 mac80211: brcm: backport remaining 5.6 kernel patches
cf2b042855 firmware-utils: add lxlfw tool for generating Luxul firmwares
cff3795450 bcm53xx: build images for Luxul ABR-4500 and XBR-4500 routers
53cd2299ee ath79: WNDR3700 v1/v2: make u-boot env partition writable
a0ca72d9ab uboot-envtools: ath79: add Netgear WNDR3700v2
5000fc53a1 ath79: fix DTS node names for Ubiquiti XW partitions
b6c01fec92 hostapd: remove erroneous $(space) redefinition
2d21357b65 ath79: ar93xx/qca95xx: move gmac/wmac/pcie node out of apb bus
95d5cbdec3 ath79: add wmac migration for all ar93xx/qca95xx SoCs
6b7eeb74db ppp: backport security fixes
085f38351f ath79: enable forceless sysupgrade from ar71xx on fritz300e
7cbd39421e ath79: add gpio4 pinmux on TL-WR841N/ND v8, WR842N v2, MR3420 v2
6a950afde1 ath79: add support for Ubiquiti Nanostation Loco M (XM)
8fa6107aee ath79: add support for Ubiquiti Picostation M (XM)
2d3a93335a ramips: append tail to WF2881 initramfs image
21bf718b8c ath79: add support for Ubiquiti NanoStation Loco M (XW)
b2660e67f0 Revert "ath79: add support for Ubiquiti NanoStation Loco M (XW)"
c9b6bb43ce ath79: phy-ar7200-usb: adapt old behavior of arch/mips/ath79/dev-usb.c
d0c8875faf ath79: ar934x: use reset for usb-phy-analog
191822b59f ipq806x: add missing core1 voltage tolerance
bc0ca20ca9 ipq806x: fix bug in L2 cache scaling
af79c3bccc kernel: bump 4.14 to 4.14.171
9e2a1af62f uhttpd: update to latest Git HEAD
0e9e5b1553 Revert "ppp: backport security fixes"
cf118077cd ppp: backport security fixes
4edadfb997 ath79: add support for Ubiquiti NanoStation Loco M (XW)
b33cfb7eb2 ath79: add missing reset-gpios for NanoStation Loco M (XW)
65030d81f3 libubox: update to latest Git HEAD
33732f4a9c OpenWrt v19.07.2: adjust config defaults

packages:
30d0c2ee0 python: Replace utime with utimes
a76dd0635 python: Fix float byte order detection
ad50eb7c0 python3: Fix float byte order detection
36a1c0c5a python-certify: bump to 2019.11.28
265e444d3 mariadb: security bump to 10.2.31
cff7a04a6 Merge pull request #11176 from jefferyto/python-fix-float-byte-order-openwrt-19.07
f7b3d0062 Merge pull request #11184 from micmac1/19.07-maria-10.2.31
8b5c2901c nginx: backport master changes to 19.07
2c4e6a539 uwsgi: backport master changes to 19.07
266917ae5 Merge pull request #11197 from Ansuel/backport
752070f84 nextdns: Update to version 1.4.23
6e5977094 libxml2/host: revert xml2-config prefix fix
0eb7b3e4e libxml2: install xml2-config with host triplet
e73b2a0a0 Merge pull request #11203 from micmac1/19.07-xml2
da3df8f6c hcxdumptool: update to 6.0.1
00a1294d0 wsdd2: fix typos
8dffab34b adblock: update 3.8.15
24c1ebc2b Merge pull request #11216 from dibdot/19.07-2
98499ee44 Merge pull request #11214 from Andy2244/wsdd2-fix_typos-(19.07)
9a792f41c golang: Update to 1.13.7, add PKG_CPE_ID to Makefile
3c95786ac Merge pull request #11219 from jefferyto/golang-1.13.7-openwrt-19.07
ba9262f04 dnscrypt-proxy2: update to version 2.0.36
afb4bb4c2 dnscrypt-proxy2: update to version 2.0.39
f828174de hcxtools: update to 5.3.0 + changes to Makefile to include latest binaries being compiled
0263d2273 sqlite3: update to version 3.31.1
6ac743909 Merge pull request #11226 from micmac1/sqlite3-19.07
1d7cda2ed golang: Improve build isolation from user environment
f8999d963 Merge pull request #11229 from jefferyto/golang-env-fixes-openwrt-19.07
433264290 Merge pull request #11213 from adde88/openwrt-19.07
90e596693 shadow: update to 4.8.1
268ea7a78 python-importlib-metadata: add new package
da2071507 syslog-ng: listen not globally but just locally
4d16e3ae3 freeradius3: Fix proxy.conf file conflict.
c417bddda freeradius3: Enable the rlm_python and rlm_python3 modules.
3cf0c61f2 php7: update to 7.2.27
04da60f17 nano: update to 4.8
391f5f087 apr: bump to 1.7.0 & sync with master
4cba41ac1 apr-util: sync with master
623ae4614 apache: bump to 2.4.41 & sync with master
8b238c840 subversion: add libsqlite3 depend
019b8fd05 subversion: fix build failure
9e551f1b3 apache: add postinstall script
9be5706a7 Merge pull request #11245 from Robby-/openwrt-19.07-rlmpython_and_fixproxycfgconflict
f815a7416 Merge pull request #11281 from micmac1/apache-19.07-up
490685093 dnsdist: fix compilation on PIE ARM64
1a503986f samba4: update to 4.11.6, add new UCI option
f5be481a4 ksmbd: update to 3.1.3, ksmbd-tools: update to 3.2.1, add smb1 support, add avahi support package
a0c00a213 Merge pull request #11297 from Andy2244/ksmbd-update-3.1.3-(19.07)
8640d243e Merge pull request #11296 from Andy2244/samba4-update-4.11.6-(19.07)
659b1b807 boost: Fix embarassing ARC typo
8e746461f measurement-kit: update to version 0.10.9
bdca0e7bf measurement-kit: Fix compilation with uClibc-ng
c7e1106d1 nss: Fix compilation with uClibc-ng
5c856ff55 perl: Don't build InstallDev under ARC
fc5935729 perl: define $sysroot for extensions
b4febf32c liburcu: Don't build under ARC, not even InstallDev
78d7c8aef knot: Do not try to build under ARC
76947a8be boost: reintroduce uClibc-ng patch
2b15682da sshpass: add new package
403c98169 Merge pull request #11356 from neheb/boostkkk
2058e86fa zabbix: Fix compilation with uClibc-ng
1e97faf11 zabbix: remove configure patch
9d6855be1 ffmpeg: Fix pkgconfig files to be more cross compile friendly
c53055aa1 ffmpeg: Fix fdkaac patches
dbef69927 ffmpeg: update to 3.4.7
bd24f9bb3 ffmpeg: disable x86 asm for old CPU types
2f5b20987 haproxy: Update HAProxy to v2.0.13
156dd4ad0 Merge pull request #11360 from gladiac1337/haproxy-2.0.13-openwrt-19.07
12b84e6ac golang: Update to 1.13.8
3cf36f145 Merge pull request #11375 from jefferyto/golang-1.13.8-openwrt-19.07
1fbf3e179 Shorewall-core: Bump to version 5.2.3.6
7c8f5fd70 Shorewall-lite: Bump to version 5.2.3.6
4f0e5c49d Shorewall6-lite: Bump to version 5.2.3.6
481e63feb Shorewall: Bump to version 5.2.3.6
1c7b7cfdf Shorewall6: Bump to version 5.2.3.6
e29cc586f Merge pull request #11381 from wvdakker/openwrt-19.07
2c0a307ee acme: update to 2.8.5
3a4d88f5d youtube-dl: update to version 2020.1.15
7153a1f7f youtube-dl: update to version 2020.1.24
201ec1470 youtube-dl: update to version 2020.2.16
47dc62ba2 nnn: update to version 2.9
11b2c1ea1 nnn: update to version 3.0
e9ea875a1 zip: remove package
7fe207375 xl2tpd: bump to version 1.3.15
61970ece5 xl2tpd: fix building failure caused by pfc
95c72d8ab php7: update to 7.2.28
8b1101bbd gnutls: updated to 3.6.12
c3ff03a34 ocserv: updated to 0.12.6
13e8c6ffe pagekite: bump to v0.91.200218
42c7a7adb pagekite: avoid redundant information in syslog
99efce0cd pagekite: add simple websockets and update syntax

LuCI:
66213ef61 luci-app-advanced-reboot: fix faulty ip.new() invocation
4f50a4fa8 luci-mod-network: dhcp.js: fix resolvefile option depends
159219787 luci-base: validation.js: fix integer and float type validation
8b1683524 luci-lib-ip: explicitly clear scope when allocating new userdata
2910f6e6b luci-base: extend rollback timeout to 90 seconds
920433457 rpcd-mod-luci: honour dhcp leases without ip address for host hints
405f5e837 luci-base: form.js: ButtonValue: pass sid as first onclick argument
1fdef8cb6 luci-base: form.js: pass section_id to Button onclick handler
2665c5fef treewide: merge master translation updates
69445fa25 luci-base: luci.js: L.dom.data(): gracefully handle non-node argument
49999e987 luci-base: cbi.js: avoid double-initialization of dropdowns
c09e27639 luci-app-samba4: add support for new UCI option
0da47ff5c Merge pull request #3621 from Andy2244/luci-app-samba4_new-UCI-(19.07)
bd38c30a3 luci-app-ksmbd: update for 3.2.1 version (usmbd -> ksmbd.mountd)
b80daa1ab luci-app-ksmbd: support new UCI option "allow_legacy_protocols"
4dea7aedd Merge pull request #3622 from Andy2244/luci-app-ksmbd_new-UCI-(19.07)
85a6f0745 luci-mod-network, luci-mod-status: harmonize wifi signal indicators
da80dcc40 luci-mod-network: explicitely configure "none" encryption in wifi join
29c29a6b4 luci-mod-network: wifi scan: add button to start/stop refreshes
4c8f0cdcc luci-mod-network: wifi join: add option to lock bssid
62be6eff0 luci-mod-network: wifi join: allow reclaiming empty wwan networks
657407f92 treewide: resync and backport translations
087e6acbf luci-app-firewall: snats.js: fix rewrite IP validation for SNAT target
02c68f283 luci-mod-status: hide DHCP status if dnsmasq/odhcpd aren't present
98ca75e91 luci-app-opkg: gracefully handle empty config files
13dd17fca treewide: merge master translations

routing:
f1b0476 luci-app-bmx6: bugfix format to query bmx6-info
efa6e54 luci-app-bmx6: bugfix querying bmx6-info (p2)
@SvenRoederer
profiles: add some NanoStation M devices to ath79
24ad7e5 
* NanoStation Loco M (XM)
* PicoStation M (XM)
These boards have been added to upstream with a dedicated profile.
@SvenRoederer
configs: remove explicit activation of ASLR
6272624 
In c271bc0 we intended to enable PKG_ASLR, but after fixing a 
typo it turns out that this breaks some packages. Some addon kernel-packages failing with `mips-openwrt-linux-musl-ld: 
unrecognized option '-specs=include/hardened-ld-pie.specs'`.
As ASLR also increases the size of the binaries upstream reworked this option in 19cbac7d264dfca1f75849de64beb98830fbb1e4
to be enabled by default for specific binaries on targets with plenty flash.
@SvenRoederer
targets: update gerneric_ffberlin to pass CONFIG_settings check not f…
c24780e 
…ailing
@SvenRoederer SvenRoederer force-pushed the gluon-scripts branch 2 times, most recently from 8d5785a to 5cecca6 Compare March 5, 2020 23:48
@SvenRoederer SvenRoederer mentioned this pull request Mar 6, 2020
Closed
@SvenRoederer
debug: add some debug to the scripts
6810144
@SvenRoederer
targets: remove some settings leftover from gloun
4a14111
@SvenRoederer
targets: remove "kmod-usb-net" from generic_ffberlin
5cc9e0a 
this module gets selected automatically; by its depenging modules
@SvenRoederer
Makefile: have gluon-config also depend on .stamp-feeds-updated $(OPE…
46bf5db 
…NWRT_DIR)/dl
@SvenRoederer
targets/generic_ffberlin: add some more required packages
b0ecf57
@SvenRoederer
targets/generic_fberlin: keep up with renamed CONFIGs of BATMAN-adv
5bbf0ae 
51256a7
@SvenRoederer
targets/generic_ffberlin: fix "CONFIG_ATH_USER_REGD"
398223c 
this option depends on kmod-ath, which is not avail on all targets.
So use try_config() to set this option, to prevent error on check_config.
@SvenRoederer
targets/generic_ffberlin: same for "CONFIG_PACKAGE_ATH_DFS"
05f1519
@SvenRoederer
targets/generic_ffberlin: shorten package-list by moving to site-config
ce75736
@SvenRoederer
targets/generic_ffberlin: move ordinary packages form config() to pac…
382a77f 
…kages-list
Akira25
Akira25 requested changes Mar 7, 2020
View reviewed changes
Copy link
Member

@Akira25 Akira25 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally think, that the folder packages or renamed imagetypes should be named packagelists or something similar. That name would meet better the expectations on the content

@SvenRoederer
Copy link
Contributor Author

SvenRoederer commented Mar 7, 2020 via email

SvenRoederer added a commit that referenced this pull request Mar 7, 2020
@SvenRoederer
rename folder as per #765 (review)
8f6cd28
@SvenRoederer SvenRoederer mentioned this pull request Mar 7, 2020
Merged
@SvenRoederer
Copy link
Contributor Author

I personally think, that the folder packages or renamed imagetypes should be named packagelists or something similar. That name would meet better the expectations on the content

@Akira25 I put your suggestion into a separate PR #766

@SvenRoederer
patches/ffberlin: update package Makefiles after luci.mk split
6dbf8b1
@SvenRoederer
modules: update berlin_packages
da61b1b
@SvenRoederer
targets/ath79-generic: add ath10k-based boards to sync with recent list
5d9abcd 
add ath10k based boards
* tp-link-archer-c59-v1
* tp-link-archer-c7-v4
* tp-link-archer-c7-v5
* ubnt_unifiac-mesh
* ubnt_unifiac-pro
@SvenRoederer SvenRoederer mentioned this pull request Mar 8, 2020
Closed
@znrR znrR added this to voted to discuss in meeting in Freifunk Berlin Firmware Meetings Apr 7, 2020
@SvenRoederer SvenRoederer mentioned this pull request Jun 28, 2020
Closed
@SvenRoederer
Copy link
Contributor Author

Sadly I must agree to the comment #825 (comment) of @Akira25, but this PR is a working solution around issue #696. And it's the only solution I've seen so far.

@Akira25
Copy link
Member

Akira25 commented Jul 5, 2020 •
edited
Loading

I see some great advantages in this commit. Unfortunatly, I wasn't able to test out by building some images. When I just fired some plain make command, the build system asked for a site configuration. @SvenRoederer is there a readily avaiable one for Berlin?

package-src: hostapd-mini  
packages   : hostapd-mini
Makefile:75: *** No site configuration was found. Please check out a site configuration to /home/martin/test_gluon_framework/site.  Stop.

@SvenRoederer
Copy link
Contributor Author

I have a (hopefully still working) one at https://github.com/SvenRoederer/ffberlin-gluon-site.
But the my goal is to not rely on a site.conf finally.

@Akira25
Copy link
Member

Akira25 commented Jul 12, 2020

Okay, I did checkout this branch and checked out your site-configuration to the directory ./site. The build looked very promising, but stoped with that prompt:

make: *** No rule to make target '/home/martin/test_gluon_framework/configs/common.config', needed by '/home/martin/test_gluon_framework/openwrt/.config'.  Stop.

@SvenRoederer
Copy link
Contributor Author

I assume you just called make w/o an explicit goal, right?

I did not change the default goal yet. So have a look at the diff of the Makefile. Calling 'make gluon-compile' and 'make gluon-images' should do the trick.

@SvenRoederer SvenRoederer mentioned this pull request Sep 18, 2020
Merged
This was referenced Sep 26, 2020
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Akira25 Akira25 Akira25 requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
buildsystem discussion enhancement has conflicts
Projects
No open projects
Freifunk Berlin Firmware Meetings
  
voted to discuss in meeting
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@SvenRoederer @Akira25