wget --spider

[Jake]

Hi Ercolino,

This is due to --spider issuing a GET request instead of a HEAD request.

I submitted a patch on 7th August for this. Details and patch copied here:

In GNU wget[1], `--spider' first issues a HEAD request, then iff HEAD fails, issues a GET request. In Busybox wget, only a GET request is sent. According to the HTTP spec[2], Busybox wget impl is technically correct. However, it causes portability problems. Interestingly, GNU wget man page[3] does not explicitly specify HTTP method.

Attached is patch to fix. Note: this is still not totally compatible with GNU wget because it does not retry with GET if HEAD fails.

Thank you,
Jake

[1] https://git.savannah.gnu.org/cgit/wget.git/tree/src/http.c#n4304
[2] https://httpwg.org/specs/rfc7231.html#HEAD
[3] https://www.gnu.org/software/wget/manual/wget.html#index-spider


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Thursday, September 30th, 2021 at 8:59 AM, Ercolino de Spiacico <bellocarico at hotmail.com> wrote:

> Thanks Nicholas
>
> I sort of got to the same conclusion that -S is somehow referring to
>
> --spider or a sort of alias to it.
>
> There is one thing that puzzles me though, the -S will show the Server
>
> response indeed but also download the file. This latter action defeats
>
> the concept for my purpose (check RAM availability prior to download to
>
> prevent starvation).
>
> Say you're in my situation, with embedded-system's RAM overloaded you
>
> need to make a decision if a file download will fit or not, -S will
>
> provide the info but potentially already creating the damage.
>
> Is there any way to get the file size only without downloading first?
>
> I've tried appending "-O /dev/null", somehow it seems to work (file not
>
> created) but I'm not sure this is the right approach... Not triggering
>
> the file download at all would be the optimal result I suppose.
>
> busybox mailing list
>
> busybox at busybox.net
>
> http://lists.busybox.net/mailman/listinfo/busybox
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wget_spider_fix.diff
Type: text/x-patch
Size: 514 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/busybox/attachments/20210930/07217db8/attachment.bin>