[Buildroot] [PATCH 4/4] pkg-download: check hasahes for locally cached files
Yann E. MORIN
yann.morin.1998 at free.fr
Wed Dec 3 18:51:57 UTC 2014
Thomas, All,
On 2014-12-02 09:31 +0100, Thomas Petazzoni spake thusly:
> On Tue, 2 Dec 2014 00:24:09 +0100, Yann E. MORIN wrote:
> > In some cases, upstream just update their releases in-place, without
> > renaming them. When that package is updated in Buildroot, a new hash to
> > match the new upstream release is included in the corresponding .hash
> > file.
> >
> > As a consequence, users who previously downloaded that package's tarball
> > with an older version of Buildroot, will get stuck with an old archive
> > for that package, and after updating their Buildroot copy, will be greeted
> > with a failed download, due to the local file not matching the new
> > hashes.
> >
> > So, to avoid this situation, check the hashes prior to doing the
> > download. If the hashes match, consider the locally cached file genuine,
> > and do not download it. However, if the locally cached file does not
> > match the known hashes we have for it, it is promptly removed, and a
> > download is re-attempted.
>
> So in essence, from now on, at each build, we are re-checking the
> hashes, while previously they were checked only when the file was
> downloaded. Not that great for build time, but well, again maybe the
> time to check the hashes is negligible compared to the build time. And
> we can assume that a big tarball, which takes a certain time to hash,
> will also contain a lot of source code to build, so the time to
> calculate the hash is proportional to the build time of the package. So
> if you're ready to spend several minutes to build Qt, you're probably
> ready to wait a few more seconds to calculate the hash of the Qt
> tarball each time.
I just hashed a 231MiB tarball (qt-everywhere-opensource-src-4.8.6.tar.gz)
and it takes (wall-clock time, core-i5 @2.5GHz):
- cache-cold sha1 : 1.914s
- cache-hot sha1 : 0.762s
- cache-hot sha256: 1.270s
(cache-cold sha256 is not easily done, I'd have to either umount /home
or reboot...)
Needless to say this is negligible when compared to the build time of Qt
(I do not have the numbers available for now, but I doubt it would be
less than 30 seconds on my machine).
I will add those numbers in the commit log next time I respin.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
More information about the buildroot
mailing list