path: root/support
diff options
authorGravatar Yann E. MORIN <yann.morin.1998@free.fr>2014-12-11 23:52:08 +0100
committerGravatar Thomas Petazzoni <thomas.petazzoni@free-electrons.com>2014-12-11 23:59:41 +0100
commit9bb7d10eeadf4aebefde672de0b81e0d321fb5b0 (patch)
treed6c3b22fe175bd148e40059a6e53f606c3bf8d01 /support
parent9b88c604843ee176bb82d3d2185b0663837920e5 (diff)
pkg-download: check hashes for locally cached files
In some cases, upstream just update their releases in-place, without renaming them. When that package is updated in Buildroot, a new hash to match the new upstream release is included in the corresponding .hash file. As a consequence, users who previously downloaded that package's tarball with an older version of Buildroot, will get stuck with an old archive for that package, and after updating their Buildroot copy, will be greeted with a failed download, due to the local file not matching the new hashes. Also, an upstream would sometime serve us HTML garbage instead of the actual tarball we requested, like SourceForge does from time for as-yet unknown reasons. So, to avoid this situation, check the hashes prior to doing the download. If the hashes match, consider the locally cached file genuine, and do not download it. However, if the locally cached file does not match the known hashes we have for it, it is promptly removed, and a download is re-attempted. Note: this does not add any overhead compared to the previous situation, because we were already checking hashes of locally cached files. It just changes the order in which we do the checks. For the records, here is the overhead of hashing a 231MiB file (qt-everywhere-opensource-src-4.8.6.tar.gz) on a core-i5 @2.5GHz: cache-cold cache-hot sha1 1.914s 0.762s sha256 2.109s 1.270s But again, this overhead already existed before this patch. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <jacmet@uclibc.org> Cc: Gustavo Zacarias <gustavo@zacarias.com.ar> Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Diffstat (limited to 'support')
1 files changed, 5 insertions, 1 deletions
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index f0cdd735b4..cced8f6a4c 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -49,7 +49,11 @@ main() {
# If the output file already exists, do not download it again
if [ -e "${output}" ]; then
- exit 0
+ if support/download/check-hash "${hfile}" "${output}" "${output##*/}"; then
+ exit 0
+ fi
+ rm -f "${output}"
+ printf "Re-downloading '%s'...\n" "${output##*/}"
# tmpd is a temporary directory in which backends may store intermediate