Opened 6 years ago

Closed 4 years ago

Last modified 3 years ago

#55011 closed enhancement (fixed)

livecheck should have curl request a compressed response

Reported by: ryandesign (Ryan Carsten Schmidt) Owned by: dgilman (David Gilman)
Priority: Normal Milestone: MacPorts 2.7.0
Component: base Version: 2.4.2
Keywords: Cc: kurthindenburg (Kurt Hindenburg)
Port:

Description

Today I noticed that youtube-dl's livecheck was not matching anymore. The reason seems to be that, although we are not requesting one, GitHub is delivering a gzip-compressed response.

Arguably, it is a bug for GitHub to deliver a compressed response when one was not requested.

Arguably, it is a bug for curl not to decompress a compressed response it received.

Nevertheless, to avoid the problem, and also to save bandwidth, we should request a compressed response when doing livecheck. The command line flag for this is --compressed. The equivalent libcurl option is to set CURLOPT_ACCEPT_ENCODING to "". Note that before curl 7.21.6 this was called CURLOPT_ENCODING, and we should include code for that to support old versions of Mac OS X.

Change History (12)

comment:1 Changed 6 years ago by ryandesign (Ryan Carsten Schmidt)

$ curl -sO http://rg3.github.io/youtube-dl/download.html
$ file download.html 
download.html: gzip compressed data, from Unix
$ curl -sO --compressed http://rg3.github.io/youtube-dl/download.html
$ file download.html 
download.html: HTML document, UTF-8 Unicode text
$

comment:2 Changed 6 years ago by kurthindenburg (Kurt Hindenburg)

Cc: kurthindenburg added

Did something change since this ticket? I get "HTML document, UTF-8 Unicode text" for both now and livecheck appears to work.

comment:3 Changed 6 years ago by ryandesign (Ryan Carsten Schmidt)

Yes; their servers have been intermittently returning correct or incorrect responses. They said they were trying out new CDN settings, which may have contributed to the problem.

Even if their problem is now resolved, we should request a compressed response for livecheck.

comment:4 Changed 4 years ago by dgilman (David Gilman)

Owner: set to dgilman
Resolution: fixed
Status: newclosed

In 8c41ca7c752b29a6df48fc7ebf14b8a80aa43d57/macports-base (master):

Enable cURL's automatic decompression

Closes: #55011

comment:5 Changed 4 years ago by jmroot (Joshua Root)

Is there any way to change this setting from a Portfile if it needs to fetch from a misconfigured server? I've seen plenty of cases where actual gzip files get served as though they were uncompressed files being transferred with gzip encoding.

comment:6 Changed 4 years ago by jmroot (Joshua Root)

Milestone: MacPorts Future

comment:7 Changed 4 years ago by jmroot (Joshua Root)

In fact, it would be safest to keep the old behaviour for everything other than livecheck.

comment:8 Changed 4 years ago by dgilman (David Gilman)

I added a flag to the underling TCL command but I didn't think to propagate that to the livecheck. Can do a 2nd PR for that, and probably need to also add a flag to however the fetch phase works.

comment:9 in reply to:  8 Changed 4 years ago by jmroot (Joshua Root)

Replying to dgilman:

Can do a 2nd PR for that, and probably need to also add a flag to however the fetch phase works.

Thanks, that would be good.

comment:10 Changed 4 years ago by tobypeterson

Having a fair bit of trouble since I rebuilt with this change. Really need to verify that all ports fetch/checksum properly, this is a significant change in behavior.

comment:11 Changed 4 years ago by tobypeterson

Notably, at least some GNU mirrors are exhibiting the behavior jmr mentioned.

comment:12 Changed 3 years ago by jmroot (Joshua Root)

Milestone: MacPorts FutureMacPorts 2.7.0
Note: See TracTickets for help on using tickets.