Error: org.macports.fetch for port mesa returned: fetch failed

[diochnos]

I have installed macports in a custom directory in this machine. Unfortunately it can not fetch mesa.

(In another machine without a custom installation directory I had no problems with selfupdate / upgrade outdated )

Log attached

[diochnos]

main.log for mesa

[diochnos]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

It looks like mesa's distfile is hosted on an ftp server. Do you typically have problems downloading files from ftp servers? Maybe some sort of restriction on your network? Do you use or need to use a proxy server?

Our mirror servers should have mirrored the file, but our mirror servers run on a network that has problems downloading files from ftp servers, so we use a proxy server to overcome that, but the connection to the proxy server sometimes breaks down, and this might be one of those time.

[diochnos]

I honestly can not explain this.

On the same machine that has the issue with upgrade outdated:

$ which wget
/opt/macports/latest/bin/wget
$ $ wget --version
GNU Wget 1.16.3 built on darwin14.1.0.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie -psl +ssl/gnutls 

... the rest of the output has been removed ...
$ 

$ wget ftp://ftp.freedesktop.org/pub/mesa/10.4.7/MesaLib-10.4.7.tar.bz2
--2015-03-31 00:28:32--  ftp://ftp.freedesktop.org/pub/mesa/10.4.7/MesaLib-10.4.7.tar.bz2
           => ‘MesaLib-10.4.7.tar.bz2’
Resolving ftp.freedesktop.org (ftp.freedesktop.org)... 131.252.210.176
Connecting to ftp.freedesktop.org (ftp.freedesktop.org)|131.252.210.176|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pub/mesa/10.4.7 ... done.
==> SIZE MesaLib-10.4.7.tar.bz2 ... 7523147
==> PASV ... done.    ==> RETR MesaLib-10.4.7.tar.bz2 ... done.
Length: 7523147 (7.2M) (unauthoritative)

MesaLib-10.4.7.tar.bz2        100%[=================================================>]   7.17M  1.72MB/s   in 5.2s   

2015-03-31 00:28:39 (1.38 MB/s) - ‘MesaLib-10.4.7.tar.bz2’ saved [7523147]

$

Further the following also works:

$ wget http://svn.macports.org/repository/macports/distfiles/mesa/MesaLib-10.4.4.tar.bz2
--2015-03-31 00:26:31--  http://svn.macports.org/repository/macports/distfiles/mesa/MesaLib-10.4.4.tar.bz2
Resolving svn.macports.org (svn.macports.org)... 17.251.224.213
Connecting to svn.macports.org (svn.macports.org)|17.251.224.213|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7507100 (7.2M) [application/x-bzip2]
Saving to: ‘MesaLib-10.4.4.tar.bz2’

MesaLib-10.4.4.tar.bz2        100%[=================================================>]   7.16M   579KB/s   in 13s    

2015-03-31 00:26:45 (574 KB/s) - ‘MesaLib-10.4.4.tar.bz2’ saved [7507100/7507100]

$

So, it appears that I have no issues downloading from an ftp site, but then the problem is even weirder because I attempted again to perform an `upgrade outdated' and it failed again to fetch the file. Finally I also attempted

$ wget http://cjj.kr.distfiles.macports.org/mesa/MesaLib-10.4.4.tar.bz2

and this one also worked (even though the error code was 403 instead of 404 in the log). I have no idea what the can be.

[adamb2903]

Same problem here, and unlike the OP I do not use a special directory. I have used my ISP's and Google's DNS servers, just on a hunch, but no joy.

[diochnos]

I can report that a few minutes ago it worked using ​ftp://ftp.freedesktop.org/pub/mesa/10.4.7/.

It still failed in the following ones that were tried before the ftp mirror above:

--->  Attempting to fetch MesaLib-10.4.7.tar.bz2 from http://mse.uk.distfiles.macports.org/sites/distfiles.macports.org/mesa
--->  Attempting to fetch MesaLib-10.4.7.tar.bz2 from http://lil.fr.distfiles.macports.org/mesa
--->  Attempting to fetch MesaLib-10.4.7.tar.bz2 from http://nue.de.distfiles.macports.org/macports/distfiles/mesa
--->  Attempting to fetch MesaLib-10.4.7.tar.bz2 from http://fco.it.distfiles.macports.org/mirrors/macports-distfiles/mesa
--->  Attempting to fetch MesaLib-10.4.7.tar.bz2 from http://her.gr.distfiles.macports.org/mirrors/macports/mpdistfiles/mesa
--->  Attempting to fetch MesaLib-10.4.7.tar.bz2 from http://ykf.ca.distfiles.macports.org/MacPorts/mpdistfiles/mesa

I am now in the process of upgrading other ports.

Thank you

[adamb2903]

Good for you. Still not working here. Is there a way that I can download it manually and get port to use the downloaded file? If not, can some attention be paid to fixing this *please*? This has trashed all of my other updates, which are waiting on this one for over three weeks now.

[rbubley]

I was having this problem too, so I manually downloaded the file using:

sudo wget -O /opt/local/var/macports/distfiles/mesa/MesaLib-10.4.7.tar.bz2 ftp://ftp.freedesktop.org/pub/mesa/10.4.7/MesaLib-10.4.7.tar.bz2

After that, port upgrade outdated worked for me.

[diochnos]

Just to mention that the problem reappeared on my end.

I downloaded manually the latest package (mesa-10.5.4.tar.xz) and put it under

...../var/macports/distfiles/mesa/

as the person above me indicated and then port upgrade outdated worked for me as well.

[jeremyhu (Jeremy Huddleston Sequoia)]

 --->  Attempting to fetch MesaLib-10.4.7.tar.bz2 from ftp://ftp.freedesktop.org/pub/mesa/10.4.7/
 Fetching distfile failed: response reading failed

This has nothing to do with mesa. It has to do with our using an ftp server for the distfiles.

[jeremyhu (Jeremy Huddleston Sequoia)]

It looks like that "response reading failed" error is from curl, below base.

What version of curl are you using? Are you using OSX's curl or MacPorts's curl for base?

[jeremyhu (Jeremy Huddleston Sequoia)]

Try the following:

/usr/bin/curl -LO ftp://ftp.freedesktop.org/pub/mesa/10.4.7/MesaLib-10.4.7.tar.bz2
/opt/local/bin/curl -LO ftp://ftp.freedesktop.org/pub/mesa/10.4.7/MesaLib-10.4.7.tar.bz2

Do either of those fail?

What version of OSX do you have?

I'm not sure if there's a way to workaround this in base if curl is having issues below us.

[diochnos]

Which returns OSX's curl. I am running OS X Yosemite 10.10.3. In any case, I can fetch the file using either curl.

$ which curl
/usr/bin/curl
$ /usr/bin/curl --version
curl 7.37.1 (x86_64-apple-darwin14.0) libcurl/7.37.1 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz 
$ /opt/macports/latest/bin/curl --version
curl 7.42.0 (x86_64-apple-darwin14.3.0) libcurl/7.42.0 OpenSSL/1.0.2a zlib/1.2.8 libidn/1.29
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets 
$ uname -a
Darwin SomeHostNameHere 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64
$

[rbubley]

Replying to jeremyhu@…:

/usr/bin/curl -LO ftp://ftp.freedesktop.org/pub/mesa/10.4.7/MesaLib-10.4.7.tar.bz2

Just sits there, counting up "Time Spent"; I CTRL-Cd it after 4 minutes of nothing else happening.

$ /usr/bin/curl --version
curl 7.30.0 (x86_64-apple-darwin13.0) libcurl/7.30.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz 

Using MacPorts curl also just sat there counting up "Time Spent".

$ /opt/local/bin/curl --version
curl 7.42.0 (x86_64-apple-darwin13.4.0) libcurl/7.42.0 OpenSSL/1.0.2a zlib/1.2.8 libidn/1.29
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets 

What version of OSX do you have?

10.9.5

[rbubley]

Cc Me!

[rbubley]

Since it was working with wget but not curl, I looked to see what ftp commands they were both sending:

Curl was "Connected to ftp.freedesktop.org (131.252.210.176)" and was then sending:

USER anonymous
PASS ftp@example.com
PWD
CWD pub
CWD mesa
CWD 10.4.7
EPSV

...and then it waited for 5 minutes, and then

421 Timeout. 0     0    0     0      0      0 --:--:--  0:05:00 --:--:--     0< 229 Entering Extended Passive Mode (|||12853|)
* Connection time-out
* Failed EPSV attempt. Disabling EPSV
> PASV
  0     0    0     0    0     0      0      0 --:--:--  0:05:01 --:--:--     0* response reading failed
* Closing connection 0

wget was sending:

USER anonymous
PASS -wget@
SYST
PWD
TYPE I
CWD /pub/mesa/10.4.7
SIZE /pub/mesa/10.4.7
PASV
RETR MesaLib-10.4.7.tar.bz2

Ftping directly into ftp.freedesktop.org and issuing the same commands as curl:

...
ftp> quote EPSV
229 Entering Extended Passive Mode (|||25232|)421 Service not available, remote server timed out. Connection closed.

[larryv (Lawrence Velázquez)]

Replying to jeremyhu@…:

What version of curl are you using? Are you using OSX's curl or MacPorts's curl for base?

AFAIK base uses the system libcurl directly.

[rbubley]

Looks to me like the problem may be with the ftp server, ftp.freedesktop.org.

It would be useful for someone for whom this works to post the output of

/usr/bin/curl -v  ftp://ftp.freedesktop.org/pub/mesa/10.4.7/MesaLib-10.4.7.tar.bz2

Incidentally, if base called curl with the --disable-epsv parameter, this would be work around the problem.

[diochnos]

Here is the (partial) output on my machine:

$  /usr/bin/curl -v  ftp://ftp.freedesktop.org/pub/mesa/10.4.7/MesaLib-10.4.7.tar.bz2
* Hostname was NOT found in DNS cache
*   Trying 131.252.210.176...
* Connected to ftp.freedesktop.org (131.252.210.176) port 21 (#0)
< 220 Welcome to the freedesktop.org FTP server.
> USER anonymous
< 331 Please specify the password.
> PASS ftp@example.com
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD pub
* ftp_perform ends with SECONDARY: 0
< 250 Directory successfully changed.
> CWD mesa
< 250 Directory successfully changed.
> CWD 10.4.7
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||13908|).
* Hostname was NOT found in DNS cache
*   Trying 131.252.210.176...
* Connecting to 131.252.210.176 (131.252.210.176) port 13908
* Connected to ftp.freedesktop.org (131.252.210.176) port 21 (#0)
> TYPE I
< 200 Switching to Binary mode.
> SIZE MesaLib-10.4.7.tar.bz2
< 213 7523147
> RETR MesaLib-10.4.7.tar.bz2
< 150 Opening BINARY mode data connection for MesaLib-10.4.7.tar.bz2 (7523147 bytes).
* Maxdownload = -1
* Getting file with size: 7523147
BZh91AY&SY?v??P?????????????????? .......

where after the dots in the end we have a bunch of gibberish which I assume reflects the binary content of the file that is being downloaded.

For convenience I am placing once again the version of /usr/bin/curl that I am using (in OS X Yosemite 10.10.3):

$ /usr/bin/curl --version
curl 7.37.1 (x86_64-apple-darwin14.0) libcurl/7.37.1 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz 
$

[rbubley]

Replying to diochnos@…:

I just tried again on a machine apparently running the same software as diochnos (Yosemite 10.10.3, with the exact same version of curl).

I get the same at the start

$ /usr/bin/curl -v  ftp://ftp.freedesktop.org/pub/mesa/10.4.7/MesaLib-10.4.7.tar.bz2
* Hostname was NOT found in DNS cache
*   Trying 131.252.210.176...
* Connected to ftp.freedesktop.org (131.252.210.176) port 21 (#0)
< 220 Welcome to the freedesktop.org FTP server.
> USER anonymous
< 331 Please specify the password.
> PASS ftp@example.com
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD pub
* ftp_perform ends with SECONDARY: 0
< 250 Directory successfully changed.
> CWD mesa
< 250 Directory successfully changed.
> CWD 10.4.7
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively

then it waits for 5 minutes followed by

421 Timeout.ng Extended Passive Mode (|||36683|)
* Hostname was NOT found in DNS cache
* Connection time-out
* Failed EPSV attempt. Disabling EPSV
> PASV
* response reading failed
* Closing connection 0
curl: (56) response reading failed

So where does that leave us? It seems that there isn't a problem at the server end, and with configs that are pretty close at the client end, it seems unlikely that the problem is there, either.

According to ​http://blogs.msdn.com/b/spike/archive/2013/07/04/a-network-analysis-of-passive-epsv-or-pasv-ftp-connection.aspx, EPSV connections can be troublesome to get right in firewalls - so it may well be a bug in my router, or a problem with the firewall at my ISP. So not obviously something that macports can fix, in general.

In the specific case of mesa, the problem may be worked around by adding the following to the port file:

fetch.use_epsv      no

[larryv (Lawrence Velázquez)]

Has likely duplicate #47667.

[denpashogai@…]

Cc Me!

[denpashogai@…]

Replying to russ.bubley@…:

In the specific case of mesa, the problem may be worked around by adding the following to the port file:

fetch.use_epsv      no

Thanks Russ, confirmed that this workaround helped in my case (behind a large-ish corporate router thingy).

[ryandesign (Ryan Carsten Schmidt)]

Replying to adam.bartley@…:

Is there a way that I can download it manually and get port to use the downloaded file?

Certainly; the process is documented in ProblemHotlist#fetch-failures

[ryandesign (Ryan Carsten Schmidt)]

Replying to denpashogai@…:

Replying to russ.bubley@…:

In the specific case of mesa, the problem may be worked around by adding the following to the port file:

fetch.use_epsv      no

Thanks Russ, confirmed that this workaround helped in my case (behind a large-ish corporate router thingy).

Jeremy: should we do this? Or will this cause problems for other users?

It might be less problematic for users to switch the port to use an http mirror instead of the ftp site.
Here are some mirrors of 10.5.4: ​http://mirrors.slackware.com/slackware/slackware-current/source/x/mesa/mesa-10.5.4.tar.xz.mirrorlist
Here is a mirror of 10.6.4, to which we could update the port: ​http://ftp.osuosl.org/pub/blfs/svn/Xorg

[jmroot (Joshua Root)]

And of course fixing #45262 would mostly hide this problem from users.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #49078.

[ryandesign (Ryan Carsten Schmidt)]

I was going to commit my suggestion to use the slackware mirror, but they have since upgraded mesa to a newer version and no longer mirror the old version. So instead I committed Russ's suggestion from comment:21 and duplicate #47701 to disable EPSV mode in r140818.