aboutsummaryrefslogtreecommitdiff
path: root/package/pkg-download.mk
Commit message (Collapse)AuthorAgeFilesLines
* core: rename FOO_BASE_NAME to FOO_BASENAME to avoid clashesGravatar Thomas De Schampheleire2018-04-081-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In current Buildroot, clashes occur between the variables _NAME and _BASE_NAME for two packages called foo and foo-base, i.e. Package foo: FOO_NAME = foo FOO_BASE_NAME = foo-1.2.3 Package foo-base: FOO_BASE_NAME = foo-base FOO_BASE_BASE_NAME = foo-base-4.5.6 where variable FOO_BASE_NAME is clashing between these two packages. Specific cases where this clash is already existing are: - alljoyn-base - alljoyn-tcl-base - perl-xml-sax-base The problem is generic and can occur for a number of variables in Buildroot. A non-exhaustive list: <pkg>_BASE and <pkg>_BASE_NAME <pkg>_BASE_NAME and <pkg>_RAW_BASE_NAME <pkg>_DIR and <pkg>_DL_DIR <pkg>_VERSION and <pkg>_DL_VERSION <pkg>_SOURCE and <pkg>_TARGET_SOURCE <pkg>_INSTALL_IMAGES and <pkg>_TARGET_INSTALL_IMAGES (same for _STAGING and _TARGET) <pkg>_LICENSE_FILES and <pkg>_MANIFEST_LICENSE_FILES <pkg>_DEPENDENCIES and <pkg>_FINAL_DEPENDENCIES One solution is to use another separator than '_' to separate the package name from the rest of the variable name. For example, a double underscore: FOO__NAME FOO__BASE_NAME FOO_BASE__NAME FOO_BASE__BASE_NAME However, making that change for only this case means that the variable naming is no longer consistent. And making the change for all variables has a large impact, also on certain user scripts. For now, keep it simple, and rename FOO_BASE_NAME into FOO_BASENAME, so that the variables become: FOO_NAME FOO_BASENAME FOO_BASE_NAME FOO_BASE_BASENAME For consistency, also adapt FOO_RAW_BASE_NAME. Since FOO_RAW_BASENAME would still pose a conflict with a package called 'foo-raw', take the opportunity to rename it into FOO_BASENAME_RAW instead, which does not pose a conflict as we have no variable called FOO_RAW. Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Reviewed-by: Sam Voss <sam.voss@rockwellcollins.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com> (cherry picked from commit 83d2644b1197564358b6cd87b2f221d79671b5cc) Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-{download, generic}: remove source-checkGravatar Maxime Hadjinlian2018-01-081-44/+0
| | | | | | | | | | | | This feature is not used by anyone in the core developpers and makes a drastic simplification of the pkg-download infrastructure harder. The future patch will move much of what's in the current pkg-download.mk file into the dl-wrapper which is a shell script. Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* Revert "pkg-download: remove explicit PKG_VERSION from github helper"Gravatar Yann E. MORIN2017-11-141-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit 1e5a8916b2ab4c9c99548fa6fbd4855eee323881. The idea was that the version string can be derived because we know the package name. However, this patch does not account for the fact that $(pkgname) always points to the latest pacakge scanned, which in all other situation we're using it, is the current package, because it is used inside one ot he xxx-inner macros that are $(eval)ualed. So $(pkgname) is evaluated "early" and gets the expected value. However, the github value is not in one of those macros, so it gets evaluated "late", when doing the actual download. So, by that time, $(pkgname) will expand to the last package scanned, which is actuall the manual (without a br2-external tree). That would require that the _SITE variable be assigned with the := assignment operator. This is weird, because that would make it the only variable to require that, but only when using the github helper, which is even less obvious and would cause a lot of trouble... The obvious fixup would seem to be to use $(PKG) instead, because that already contains the upper-case package name that vcan be used as a prefix to variables. However, that does not work either, because we have a check that forbids a trsailing slash in _SITE, check that is done in pacakge/pkg-generic, inside the xxx-inner macro, during the $(eval) call. And at that time, PKG is not yet defined, because it is only defined for an actual recipe. So we can't seem to have a workable solution. So, just revert the patch. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Marcin Nowakowski <marcin.nowakowski@imgtec.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-download: remove explicit PKG_VERSION from github helperGravatar Marcin Nowakowski2017-09-241-2/+2
| | | | | | | | | | | | | | | | Currently it is always required to add package version as an argument to the github helper. Since the version is always defined as PKG_VERSION, drop this argument and generate it automatically inside the helper routine. The github helper function is extended to support both 2 and 3 argument variants (ie. either use the provided package version argument or automatically substitute with PKG_VERSION if not available), which can make the transition of the package files easier as well allows using the 3-argument variant outside of package definitions. Signed-off-by: Marcin Nowakowski <marcin.nowakowski@imgtec.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* pkg-download: enable hash check for git downloadsGravatar Arnout Vandecappelle2017-06-111-1/+2
| | | | | | | | For a while already, our git downloaded tarballs are reproducible, so we can actually check the hash for them. Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: fix fetching URLs with ? from PRIMARY/BACKUPGravatar Arnout Vandecappelle2016-10-231-2/+6
| | | | | | | | | | | | | | | | | | | Some packages download files (especially patches) with a ? in the URL. The ? marks the query part of the URL. However, the downloaded file still contains the ? but from then on it doesn't designate a query part anymore. Therefore, when fetching from PRIMARY or BACKUP site over http, the server will report a 404 Not Found. To fix, we need to replace the ? with %3F. Obviously, this should be done only when fetching from PRIMARY or BACKUP. For fetching from the real upstream, the ? really does designate the query part. Fixes #9371. Reported-by: Johan Derycke <johanderycke@gmail.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> [Thomas: replace 'SECONDARY' by 'BACKUP'.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: use raw basename for repo archiving to remove host- prefixGravatar Thomas De Schampheleire2016-10-201-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For packages that use a version control repository rather than a pre-made tarball, the directory prefix used inside the tarball is currently FOO_BASE_NAME, which can be 'foo' or 'host-foo'. This means that the hash of such tarball will be different for target and host packages, even though the contents are exactly the same. Hence, if the hash file is created based on 'foo', and later a fresh build is made where 'host-foo' happens to be built before 'foo' (with a different config, for example), the hash will be detected as incorrect and a new download is started. This problem does not affect many packages/users, due to the number of conditions to be met: - the package should be available for target _and_ host - the package needs to use a VCS download method, e.g. git, hg, svn, ... This does not include standard github downloads, which download a pre-made archive. - there should be a hash file containing the hash of the downloaded archive. Since normally there is no hash file for packages with sources coming from a version control system, this restricts even further. Some examples of packages in this category that do have a hash file (but not necessarily match the earlier conditions): expedite, vexpress-firmware, squashfs, ... - the archive needs to be stored in a 'primary site' after initial archiving and thus be downloaded later using a non-version-controlled method, like wget or scp. This is because the version control download methods do not receive a '-H' parameter pointing to the hash file and thus no hashes are checked at all even if the file is present. While packages matching the third condition could be considered to be 'wrong' and need to be fixed, it does actually makes sense to have a hash file for packages from version control, in particular if they are stored in a primary site as mentioned in the last condition. Regardless of any different opinions on the previous paragraph, it is also not conceptually correct that a tarball of a package source can contain a Buildroot-specific directory prefix 'host-'. Therefore, use FOO_RAW_BASE_NAME instead of FOO_BASE_NAME when calling the dl-wrapper. Example test scenario that exhibits the problem: $ rm -rf /tmp/dl dl/squashfs-9c1db6d13a51a2e009f0027ef336ce03624eac0d.tar.gz $ make qemu_x86_64_defconfig $ make host-squashfs-dirclean host-squashfs-source $ mkdir /tmp/dl $ mv dl/squashfs-9c1db6d13a51a2e009f0027ef336ce03624eac0d.tar.gz /tmp/dl/ $ sed -i -e 's,BR2_PRIMARY_SITE=.*,BR2_PRIMARY_SITE="file:///tmp/dl",' \ -e '/BR2_PRIMARY_SITE/aBR2_PRIMARY_SITE_ONLY=y' .config $ make host-squashfs-dirclean host-squashfs-source Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: Allow packages to pass generic options to download methodsGravatar Romain Perier2016-08-231-8/+16
| | | | | | | | | | | | Introduce a new package variable $(PKG)_DL_OPTS. When this variable is defined, its value is passed to the downloader as options to the underlying command. Packages can now retrieve archives from server expecting logins and passwords, use referer url, proxy or specific options for cloning a repository. Signed-off-by: Romain Perier <romain.perier@free-electrons.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* core/pkg-infra: download git submodules if the package wants themGravatar Yann E. MORIN2016-07-021-0/+1
| | | | | | | | | | | | | | | Add a new package variable that packages can set to specify that they need git submodules. Only accept this option if the download method is git, as we can not get submodules via an http download (via wget). Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Aleksandar Simeonov <aleksandar@barix.com> Tested-by: Matt Weber <matt@thewebers.ws> Reviewed-by: Matt Weber <matt@thewebers.ws> Tested-By: Nicolas Cavallari <nicolas.cavallari@green-communications.fr> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* core/pkg-download: ignore hashes from mirror for VCS downloadsGravatar Yann E. MORIN2016-04-201-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a download via a VCS method fails, Buildroot attempts the download from the backup mirror (if any is set). Such a download is done with the wget helper. Given a package that has a .hash file for relase tarballs, which also allows the user to use a random revision from a VCS, the normal download wrapper will not check for hashes, on the assumption that maybe they are not reproducible [*] (or user configurable). However, when the download fails (for any reason: network error, upstream removed the revision, old machine with incomplete set of CA certificates...), the backup download now proceeds over with http (or https) with the wget wrapper. The wget wrapper *always* checks for hashes, and when a .hash file is present but no hash is found for the download, this is considered an error. However, when we download from the backup mirror for a main download that should have been done with git/hq/svn/.., we in this situation have no hash for the download. Thus, we should not fail on a missing hash for that download. Same situation if a primary site is used. Add a test for the site-method in the DOWNLOAD_INNER macro; if it was either one of the VCS method, pass a one-off BR_NO_CHECK_HASH_FOR variable set to contain the tarball to download. Fixes issues like those reported by the Travis build bots: https://travis-ci.org/buildroot/buildroot-defconfig-testing/jobs/123624879 [Peter: Move logic to beginning of DOWNLOAD_INNER so it also applies for BR2_PRIMARY_SITE] Reported-by: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-download: do not test SITE_METHODGravatar Jérôme Pouiller2015-12-181-6/+1
| | | | | | | | | SITE_METHOD is always set in inner-generic-package. So, it is useless to test it in pkg-download. Signed-off-by: Jérôme Pouiller <jezz@sysmic.org> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: support 'file://' in BR2_PRIMARY_SITEGravatar Jérôme Pouiller2015-11-191-0/+1
| | | | | | | | | | | | | | | | | This feature was already discuted here: http://lists.busybox.net/pipermail/buildroot/2015-April/125419.html Personally, I have a big central directory with all tarballs I have ever downloaded. I use this feature to isolate tarballs necessary to build a configuration: make project_defconfig make BR2_DL_DIR=/tmp/mirror-project BR2_PRIMARY_SITE=file:///home/user/dl source tar -C /tmp -czvf mirror-project.tgz mirror-project Signed-off-by: Jérôme Pouiller <jezz@sysmic.org> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/download: relocate the git clone commentGravatar Baruch Siach2015-05-311-6/+0
| | | | | | | | | | | | | Following commit 95a572282e87 (pkg-infra: move the git download helper to a script, 2014-07-02), move the comment describing the shallow clone trickery as well. Merge this comment with the existing helper comment that was added in 7e40a1103a91 (support/download: convert git to use the wrapper, 2014-08-03). Rename $($(PKG)_DL_VERSION) to ${cset} to match the helper code context. Cc: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Baruch Siach <baruch@tkos.co.il> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-infra: assign BR_NO_CHECK_HASH_FOR so it is recursively-expandedGravatar Yann E. MORIN2015-05-021-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, assigning to BR_NO_CHECK_HASH_FOR but referencing variable assigned to later, fails. Here's a failing test-case, which is the reduced test-case of how we handle BR_NO_CHECK_HASH_FOR for now (warning added for test-case): export BR_NO_CHECK_HASH_FOR BR_NO_CHECK_HASH_FOR += $(XENOMAI_SOURCE) XENOMAI_VERSION = 2.6.4 XENOMAI_SOURCE = xenomai-$(XENOMAI_VERSION).tar.bz2 $(warning BR_NO_CHECK_HASH_FOR='$(BR_NO_CHECK_HASH_FOR)') all: Run it with simply make: $ make BR_NO_CHECK_HASH_FOR=' ' Now, change the first line to read: export BR_NO_CHECK_HASH_FOR = And we now get: $ make BR_NO_CHECK_HASH_FOR=' xenomai-2.6.4.tar.bz2' This new behaviour will be needed later for Xenomai, which handles the version string in an unusual way, so we can exclude its custom versions. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: fix indentation for SOURCE_CHECK_* macrosGravatar Thomas Petazzoni2015-04-261-5/+5
| | | | | | | | | | | | Some of the SOURCE_CHECK_* macros are using a non-standard two-spaces indentation. This commit switches them to use a single tab based indentation, like in the rest of Buildroot. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Tested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> [with git show -w] Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
* pkg-download: get rid of DL_MODEGravatar Thomas Petazzoni2015-04-261-5/+1
| | | | | | | | | | | The DL_MODE variable is now no longer used with any other value than "DOWNLOAD", so it no longer makes sense to have this variable at all. Therefore, this commit gets rid of it. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Tested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
* pkg-download: extend DOWNLOAD_INNER, add a SOURCE_CHECK macroGravatar Thomas Petazzoni2015-04-261-12/+16
| | | | | | | | | | | | | | | | | | | | | As part of moving to a package infrastructure based source-check implementation, we are going to move away from the global DL_MODE variable to select the behavior of the DOWNLOAD_INNER macro. As a preparation to this, this commit makes the DOWNLOAD_INNER macro take a third argument, which is the action to be done: either DOWNLOAD or SOURCE_CHECK. For now, the DOWNLOAD macro passes $(DL_MODE) as this third argument, in order to keep the existing behavior. In addition, a SOURCE_CHECK macro is added, which calls DOWNLOAD_INNER with the appropriate action. This macro will be used in the upcoming package infra based implementation of source-check. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr> [tested by doing a "make source" on a randpackageconfig] Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* pkg-download: remove support for the SHOW_EXTERNAL_DEPS DL_MODEGravatar Thomas Petazzoni2015-04-261-43/+2
| | | | | | | | | | | | Now that the external-deps implementation relies on the per-package <pkg>-all-external-deps and <pkg>-external-deps targets and no longer on the 'source' target with a custom DL_MODE, we can get rid of the support for the SHOW_EXTERNAL_DEPS DL_MODE value in the pkg-download logic. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/download: add possibility to not fail on missing hashGravatar Yann E. MORIN2015-04-251-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In very constrained cases, it might be needed to not fail if a hash is missing. This is notably the case for custom external toolchains to be downloaded, because we do have a .hash file for external toolchains, but we obviously can not have hashes for all existing custom toolchains (he, "custom"!). So, add a way to avoid failing in that case. >From the Makefile, we export the list of files for which not to check the hash. Then, from the check-hash script, if no check was done, and the file we were trying to match in in this exclusion list, we just exit without error. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Tested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> changes v6 -> v7: - /beautify/ the pattern in the case clause Changed v5 -> v6: (Arnout) - fix the pattern in the case clause Changes v4 -> v5: - micro-optimisation, use case-esac instead of a for-loop (Arnout) - typoes (Arnout) Changes v3 -> v4: - drop the magic value, use a list of excluded files (Arnout) Changes v1 -> v2: - fix typoes in commit log Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Tested-by: Gustavo Zacarias <gustavo@zacarias.com.ar> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* package infra: do not check hashes when downloading from a repositoryGravatar Yann E. MORIN2015-04-031-5/+0
| | | | | | | | | | | | | When downloading from a repository, we have no way to ensure the reproducibility of the generated archives, so we can't check the hashes. Do not specifiy a hash file in those cases. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* package/pkg-download: do not hard-code QUIETGravatar Yann E. MORIN2015-01-041-4/+4
| | | | | | | | | | | Do not hard-code QUIET in our download commands, since it is handled in the backends. Suggested by Fabio. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Fabio Porcedda <fabio.porcedda@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: silence downloads if make is silentGravatar Fabio Porcedda2015-01-041-0/+8
| | | | | | | | | | | If doing a silent build (make -s -> QUIET=-q), silence all downloads, by passing the -q flag downward to backends as well as to check-hash. Change a printf to use the trace functions. Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: verify the hashes from the download wrapperGravatar Yann E. MORIN2014-12-111-17/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of repeating the check in our download rules, delegate the check of the hashes to the download wrapper. This needs three different changes: - add a new argument to the download wrapper, that is the full path to the hash file; if the hash file does not exist, that does not change the current behaviour, as the existence of the hash file is checked for in the check-hash script; - add a third argument to the check-hash script, to be the basename of the file to check; this is required because we no longer check the final file with the final filename, but an intermediate file with a temporary filename; - do the actual call to the check-hash script from within the download wrapper. This further paves the way to doing pre-download checks of the hashes for the locally cached files. Note: this patch removes the check for hashes for already downloaded files, since the wrapper script exits early. The behaviour to check localy cached files will be restored and enhanced in the following patch. [Thomas: fix minor typo in comment.] Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <jacmet@uclibc.org> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: check for already downloaded file in the download wrapperGravatar Yann E. MORIN2014-12-111-8/+0
| | | | | | | | | | | | | | | | Instead of repeating the same test again and again in all our download rules, just delegate the check for an already downloaded file to the download wrapper. This clears up the path for doing the hash checks on a cached file before the download. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <jacmet@uclibc.org> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar> Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/download: add option parsing to the download wrapperGravatar Yann E. MORIN2014-12-111-16/+26
| | | | | | | | | | | | | | | | Instead of relying on argument ordering, use actual options in the download wrapper. Download backends (bzr, cp, hg...) are left as-is, because it does not make sense to complexify them, since they are almost very trivial shell scripts, and adding option parsing would be really overkill. This commit also renames the script to dl-wrapper so it looks better in the traces, and it is not confused with another wrapper. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: VERIFY_HASH: use same hash file for host and target downloadsGravatar Peter Korsgaard2014-12-081-3/+3
| | | | | | | | | | | Fixes (or atleast stops the build with a saner description): http://autobuild.buildroot.net/results/1a9/1a9643f8633db038d4fe5ca4a32e4d52e70a3a1a/ We're using the same sources for host and target downloads, so it makes sense to use the same <pkg>.hash file as well to ensure the host version gets verified without us having to maintain a seperate host-<pkg>.hash file. Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* check-hash: silence hash ok messages when make -s is usedGravatar Peter Korsgaard2014-10-251-1/+1
| | | | | | Similar to what we do for apply-patches.sh Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* .mk files: bulk aligment and whitespace cleanup of assignmentsGravatar Thomas De Schampheleire2014-10-071-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Buildroot coding style defines one space around make assignments and does not align the assignment symbols. This patch does a bulk fix of offending packages. The package infrastructures (or more in general assignments to calculated variable names, like $(2)_FOO) are not touched. Alignment of line continuation characters (\) is kept as-is. The sed command used to do this replacement is: find * -name "*.mk" | xargs sed -i \ -e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*$#\1 \2#' -e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*\([^\\]\+\)$#\1 \2 \3#' -e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*\([^\\ \t]\+\s*\\\)\s*$#\1 \2 \3#' -e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\(\s*\\\)#\1 \2\3#' Brief explanation of this command: ^\([A-Z0-9a-z_]\+\) a regular variable at the beginning of the line \([?:+]\?=\) any assignment character =, :=, ?=, += \([^\\]\+\) any string not containing a line continuation \([^\\ \t]\+\s*\\\) string, optional whitespace, followed by a line continuation character \(\s*\\\) optional whitespace, followed by a line continuation character Hence, the first subexpression handles empty assignments, the second handles regular assignments, the third handles regular assignments with line continuation, and the fourth empty assignments with line continuation. This expression was tested on following test text: (initial tab not included) FOO = spaces before FOO = spaces before and after FOO = tab before FOO = tab and spaces before FOO = tab after FOO = tab and spaces after FOO = spaces and tab after FOO = \ FOO = bar \ FOO = bar space \ FOO = \ GENIMAGE_DEPENDENCIES = host-pkgconf libconfuse FOO += spaces before FOO ?= spaces before and after FOO := FOO = FOO = FOO = FOO = $(MAKE1) CROSS_COMPILE=$(TARGET_CROSS) -C AT91BOOTSTRAP3_DEFCONFIG = \ AXEL_DISABLE_I18N=--i18n=0 After this bulk change, following manual fixups were done: - fix line continuation alignment in cegui06 and spice (the sed expression leaves the number of whitespace between the value and line continuation character intact, but the whitespace before that could have changed, causing misalignment. - qt5base was reverted, as this package uses extensive alignment which actually makes the code more readable. Finally, the end result was manually reviewed. Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Cc: Yann E. Morin <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* support/download: convert wget to use the wrapperGravatar Yann E. MORIN2014-08-041-1/+3
| | | | | | | | | | This drastically simplifies the wget helper, as it no longer has to deal with atomically saving the downloaded archive. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> (Tested by running 'make busybox-source') Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/download: convert svn to use the wrapperGravatar Yann E. MORIN2014-08-041-2/+5
| | | | | | | | | | This drastically simplifies the svn helper, as it no longer has to deal with atomically saving the downloaded archive. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> (Tested by running 'make open2300-source') Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/download: convert scp to use the wrapperGravatar Yann E. MORIN2014-08-041-2/+3
| | | | | | | | | | | This drastically simplifies the scp helper, as it no longer has to deal with atomically saving the downloaded archive. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> (Tested by setting a primary site to 'scp://localhost:/tmp' and running 'make vim-source') Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/download: convert Hg to use the wrapperGravatar Yann E. MORIN2014-08-041-2/+5
| | | | | | | | | | This drastically simplifies the hg helper, as it no longer has to deal with atomically saving the downloaded archive. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> (Tested by running 'make vim-source') Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/download: convert git to use the wrapperGravatar Yann E. MORIN2014-08-041-2/+5
| | | | | | | | | | This drastically simplifies the git helper, as it no longer has to deal with atomically saving the downloaded archive. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> (Tested by running 'make fmc-fsl-sdk-source') Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/download: convert cvs to use the wrapperGravatar Yann E. MORIN2014-08-041-3/+6
| | | | | | | | | This drastically simplifies the cvs helper, as it no longer has to deal with atomically saving the downloaded archive. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/download: convert localfiles to use the wrapperGravatar Yann E. MORIN2014-08-041-2/+3
| | | | | | | | | | This drastically simplifies the localfiles helper, as it no longer has to deal with atomically saving the downloaded archive. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> (Tested by setting BUSYBOX_SITE = file:///tmp and running 'make busybox-source') Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/download: convert bzr to use the wrapperGravatar Yann E. MORIN2014-08-041-1/+5
| | | | | | | | This drastically simplifies the bzr helper, as it no longer has to deal with atomically saving the downloaded archive. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-download: Fix DOWNLOAD_LOCALFILESGravatar Andreas Naumann2014-07-181-1/+1
| | | | | | | | The cp helperscript expects a filename as target, but just the folder name was given. Signed-off-by: Andreas Naumann <anaumann@ultratronik.de> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
* pkg-infra: add possiblity to check downloaded files against known hashesGravatar Yann E. MORIN2014-07-041-3/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | Some of the packages that Buildroot might build are sensitive packages, related to security: openssl, dropbear, ca-certificates... Some of those packages are downloaded over plain http, because there is no way to get them over a secure channel, such as https. In these dark times of pervasive surveillance, the potential for harm that a tampered-with package could generate, we may want to check the integrity of those sensitive packages. So, each package may now provide a list of hashes for all files that needs to be downloaded, and Buildroot will just fail if any downloaded file does not match its known hash, in which case it is removed. Hashes can be any of the md5, sha1 or sha2 variants, and will be checked even if the file was pre-downloaded. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Baruch Siach <baruch@tkos.co.il> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-infra: move the cp (aka localfiles) download helper to a scriptGravatar Yann E. MORIN2014-07-031-2/+3
| | | | | | | | Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-infra: move the scp download helper to a scriptGravatar Yann E. MORIN2014-07-031-2/+3
| | | | | | | | Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-infra: move the bzr download helper to a scriptGravatar Yann E. MORIN2014-07-031-2/+2
| | | | | | | | Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-infra: move the wget download helper to a scriptGravatar Yann E. MORIN2014-07-031-10/+3
| | | | | | | | | Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-infra: move the hg download helper to a scriptGravatar Yann E. MORIN2014-07-021-8/+3
| | | | | | | | | | | | Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com> [tested a particular scenario that used to fail, when the 'hg archive' step is interrupted, now working fine] Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-infra: move the cvs download helper to a scriptGravatar Yann E. MORIN2014-07-021-7/+4
| | | | | | | | | Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-infra: move the svn download helper to a scriptGravatar Yann E. MORIN2014-07-021-6/+3
| | | | | | | | | | | Maintaining the download helpers in the Makefile has proved to be a bit complex, so move it to a shell script. [Peter: redirect pushd/popd output to /dev/null] Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Reviewed-by: Ryan Barnett <ryan.barnett@rockwellcollins.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-infra: move the git download helper to a scriptGravatar Yann E. MORIN2014-07-021-14/+3
| | | | | | | | | | | | | | | | | | | The git download helper is getting a bit more complex. Fixing it in the Makefile when it breaks (like the recent breakage with a non-existing sha1-cset) proves to be challenging, to say the least. Move it into a shell script in support/download/git, which will make it much easier to read, maintain, fix and enhance in the future. [Peter: redirect pushd/popd output to /dev/null] Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Peter Korsgaard <jacmet@uclibc.org> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Acked-by: Luca Ceresoli <luca@lucaceresoli.net> Cc: Arnout Vandecappelle <arnout@mind.be> Reviewed-by: Samuel Martin <s.martin49@gmail.com> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-download: fix svn download for deleted pathGravatar Ryan Barnett2014-06-291-2/+2
| | | | | | | | | | | | | | | | If a SVN path has been deleted in a future revision, downloading a package's source will fail using 'svn checkout -r'. Fix using URL@REV syntax since the SVN command will then verify the path at the specified version. Without using the URL@REV syntax, the SVN command will attempt to verify the path exists on the the HEAD revision before checking out the path at the specified revision. Error seen if SVN path has been deleted in HEAD using svn checkout -r: svn: E160013: '/svn/path/' path not found Signed-off-by: Ryan Barnett <ryan.barnett@rockwellcollins.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* replace github with GitHubGravatar Simon Dawson2014-06-041-1/+1
| | | | | | | | The correct capitalisation pattern is "GitHub"; fix manual and makefile commentary. Signed-off-by: Simon Dawson <spdawson@gmail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* pkg-download: update the github helperGravatar Samuel Martin2014-04-061-1/+1
| | | | | | | | | | | | | | Once again, github updates the source download url. Even if only the zip archive link is advertised on the repositories' page, the *.tar.gz is still available. It is worthy to note that the tarball's content differs depending if it has been fetched from the former and the new url (the root directory name changes). Signed-off-by: Samuel Martin <s.martin49@gmail.com> Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pkg-infra: DOWNLOAD is never called with two argumentsGravatar Yann E. MORIN2014-03-021-3/+2
| | | | | | | | | | | | Currently, the DOWNLOAD helper accepts one or two arguments: - the source URI - the source filename; if missing, it is derived from the URI But we have no caller of DOWNLOAD that passes a second argument. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>