wget --spider

[Nicholas Niro]

Hey there Ercolino,

On Wed, Sep 29, 2021 at 07:45:22PM +0200, Ercolino de Spiacico wrote:
> It would be great if busybox would allow wget to have the --spider option
> made available. --spider specifically provides the length value which is the
> size of the file we are willing to download.

It seems that busybox already have '--spider' support. According to
the GNU man page (at least the interpretation I'm doing) is that
--spider is to be used only to check the disponibility of a file.
Technically, the part that GNU wget outputs the size of the file is
extra.

That said, there is a technique that could be used with busybox wget
to actually get the length of a file before deciding to download it.
We can use the '-S' option to get the values of the HTTP header for the
file (which usually contains the file size).

I made a small one liner to get the size of a file for you (and it uses
--spider btw, which ensures we don't actually download the file):

busybox wget --spider -S https://example.com 2>&1 | busybox sed -ne '/Length/ {s/ *Content-Length: *\([0-9]\+\)/\1/ ; p}'

The value we are after should be "Content-Length".

That should enable you to implement what you're after :)

> 
> This is particularly helpful in embedded devices such as opensource
> router-firmware (Tomato, DD-WRT, OpenWRT, etc.) where there's no permanent
> filesystem and a squash filesystem to deal with. Often files are to be
> downloaded from Internet and end up on a filesystem hosted in RAM. So it
> would be priceless to have an option such as wget --spider to check the
> length (size) of the file to download so that a control can be implemented
> on whether to download or not based on RAM availability.
> 
> Thanks

Thanks

> _______________________________________________
> busybox mailing list
> busybox at busybox.net
> http://lists.busybox.net/mailman/listinfo/busybox
>