openssl 1.0.1 breaks multiple ports

[pixilla@…]

After activating openssl @1.0.0g_0 and rebuilding curl this command works where as after the openssl @1.0.1_1 upgrade it did not:

$ curl https://api-aa-3t.paypal.com/2.0/

[raimue@…]

What do you mean by "did not work"? What did you expect? What actually happened? What error message do you see?

[ryandesign@…]

I'm not familiar with the option "-14k" but I get:

$ curl -14k https://api-aa-3t.paypal.com/2.0
curl: (35) Unknown SSL protocol error in connection to api-aa-3t.paypal.com:443

Or trying verbosely without that option:

$ time curl -v https://api-aa-3t.paypal.com/2.0
* About to connect() to api-aa-3t.paypal.com port 443 (#0)
*   Trying 66.211.168.195...
* connected
* Connected to api-aa-3t.paypal.com (66.211.168.195) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to api-aa-3t.paypal.com:443 
* Closing connection #0
curl: (35) Unknown SSL protocol error in connection to api-aa-3t.paypal.com:443 

real	1m0.175s
user	0m0.048s
sys	0m0.021s

Is that what you're seeing as well? If so, I doubt it's a MacPorts-specific problem, since all we did was update the openssl port to version 1.0.1. Have you reported the problem to the developers of openssl / curl / paypal? (I'm not sure which is at fault.)

Trying to connect to ​https://trac.macports.org/ instead works fine.

[dave@…]

This isn't just a problem with curl. The new openssl breaks HTTPS access from ruby.

[dave ~] port installed openssl
The following ports are currently installed:
  openssl @1.0.0e_1
  openssl @1.0.1_0
  openssl @1.0.1_1 (active)
[dave ~] curl -v https://graph.facebook.com/
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 69.171.234.34...
* connected
* Connected to graph.facebook.com (69.171.234.34) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
[long pause]
* Unknown SSL protocol error in connection to graph.facebook.com:443 
* Closing connection #0
curl: (35) Unknown SSL protocol error in connection to graph.facebook.com:443 
[dave quane:~] ruby -ropen-uri -e 'open("https://graph.facebook.com/") &:read'
[long pause]
/opt/local/lib/ruby/1.8/net/http.rb:586:in `connect': Connection reset by peer - SSL_connect (Errno::ECONNRESET)
	from /opt/local/lib/ruby/1.8/net/http.rb:586:in `connect'
	from /opt/local/lib/ruby/1.8/net/http.rb:553:in `do_start'
	from /opt/local/lib/ruby/1.8/net/http.rb:542:in `start'
	from /opt/local/lib/ruby/1.8/open-uri.rb:242:in `open_http'
	from /opt/local/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
	from /opt/local/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
	from /opt/local/lib/ruby/1.8/open-uri.rb:162:in `catch'
	from /opt/local/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
	from /opt/local/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
	from /opt/local/lib/ruby/1.8/open-uri.rb:518:in `open'
	from /opt/local/lib/ruby/1.8/open-uri.rb:30:in `open'
	from -e:1
[dave ~] sudo port activate openssl@1.0.0e_1
--->  Computing dependencies for openssl
--->  Deactivating openssl @1.0.1_1
--->  Cleaning openssl
--->  Activating openssl @1.0.0e_1
--->  Cleaning openssl
[dave ~] curl -v https://graph.facebook.com/
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 69.171.228.43...
* connected
* Connected to graph.facebook.com (69.171.228.43) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
[curl goes on to connect and get a 302, which is correct, and there is no long pause]
[dave quane:~] ruby -ropen-uri -e 'open("https://graph.facebook.com/") &:read'
/opt/local/lib/ruby/1.8/open-uri.rb:174:in `open_loop': redirection forbidden: https://graph.facebook.com/ -> http://developers.facebook.com/docs/api (RuntimeError)
	from /opt/local/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
	from /opt/local/lib/ruby/1.8/open-uri.rb:518:in `open'
	from /opt/local/lib/ruby/1.8/open-uri.rb:30:in `open'
	from -e:1
[this is the correct behavior, and there is no long pause]

[nonstop.server@…]

Cc Me!

[mww@…]

curl works like a charme gere -- cant reproduce you problem;

[rainer@…]

Replying to pixilla@…:

After activating openssl @1.0.0g_0 and rebuilding curl this command works where as after the openssl @1.0.1_1 upgrade it did not:

$ curl -14k https://api-aa-3t.paypal.com/2.0/

mutt stopped working with SSL as well. Seems not curl, but openssl is broken, others have similar problems, see for example ​https://bbs.archlinux.org/viewtopic.php?pid=1076140 If indeed openssl is the problem, I'd suggest changing the priority to highest.

Rainer

[rainer@…]

Cc Me!

[ryandesign@…]

Rainer, as it says in that thread you linked to, and as I said above in this ticket, report the problem to the developers of OpenSSL. Once they fix it, we'll be happy to incorporate the fix in MacPorts.

[pixilla@…]

Sorry, I was to busy fixing openssl breakage to file a better ticket. Paypals sandbox endpoint works with the new openssl but their live endpoint does not. So sad.

$ curl https://api-aa-3t.paypal.com/2.0/
curl: (35) Unknown SSL protocol error in connection to api-aa-3t.paypal.com:443 

$ curl -s --connect-timeout 3 https://api-aa.sandbox.paypal.com/2.0/ | cut -c3-19
xml version="1.0"

Both of these commands work with openssl @1.0.0g_0. Should we downgrade openssl and bump the epoch?

[ryandesign@…]

We should report the problem to the developers of OpenSSL and give them a chance to respond first.

[pixilla@…]

Replying to ryandesign@…:

We should report the problem to the developers of OpenSSL and give them a chance to respond first.

A broken OpenSSL is disastrous. I don't understand why this is not an immediate rollback while we wait to hear from the developer.

[ryandesign@…]

It seems not to be broken in all circumstances, nor even for all users, according to what mww wrote above.

If we were to roll back openssl, recall that we'll also have to again increase the revision of the ports whose revisions we increased earlier to force them to rebuild against the new version.

[pixilla@…]

mww: On 10.5 PPC and 10.6 Intel this curl command fails with openssl1.0.1 and works with openssl 1.0.0g. Can you try this command on your system and if it works help me find between our systems?

$ curl https://api-aa-3t.paypal.com/2.0/

[mww@…]

I guess that the inline-assembler breaks: I enabled it for 1.0.1 as all tests passed on 10.7/x64 -- what they did not before.

I disabled "asm" for 10.5 and 10.6 as this seems to be a PPC _or_ OS-version problem; so 10.5/ppc and 10.6/ppc do not work.
What OS-version and arch is everyone else on? I would like to complete my list:

10.7/x65: asm
10.6/ppc: no-asm
10.5/ppc: no-asm

[ryandesign@…]

There is no such animal as 10.6/ppc; 10.6 runs on Intel processors only.

[ryandesign@…]

Perhaps then whether or not inline assembly works is dependent not on the OS version or processor architecture but the compiler or compiler version.

[ryandesign@…]

openssl @1.0.1_1 10.6 all tests pass

[ryandesign@…]

I don't think the asm was the problem. It still fails in the same way for me on 10.6 after updating to 1.0.1_2. Also, on 1.0.1_1 with asm on, all the tests passed for me too.

[pixilla@…]

1.0.1_2 still fails on 10.6 intel.

[dports@…]

Yeah, I'm seeing the same thing on Lion/64 regardless of whether --enable-asm is used or not.

This sounds like ​http://rt.openssl.org/Ticket/Display.html?id=2771 (use guest/guest to log in) which suggests it might be a problem with the servers rather than openssl -- but we may still want to consider going back to 1.0.0 anyway.

[pixilla@…]

Thanks Dan. Adding --sslv3 makes a difference in my curl example.

$ curl -s --sslv3 https://api-aa-3t.paypal.com/2.0/ | cut -c3-19
xml version="1.0"

Still, looks like there is a bug in openssl tls support.

[william@…]

Me Too™.

I find that when using libcurl to make requests to a server over https, these requests succeed when using OpenSSLv1.0.0g but fail with 1.0.1. Note that they also fail if I use the +gnutls variant of curl, which may indicate that it is indeed a problem with the server rather than with OpenSSL.

I expect the problem is with the length of the “client hello” being greater than one byte, as mentioned in the openssl ticket above. If I force TLSv1 (which curl allows me to do) my fetches work OK. I expect this reduces the length of the “client hello” and that allows things to work.

So my feeling is that it is not necesary to roll OpenSSL back, rather to request owners of broken servers to unbreak them, and otherwise to work around the problem by specifying a single cipher using e.g. (for curl) “curl_easy_setopt (_, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1)” and equivalents for other packages.

[william@…]

Replying to william@…:

I expect the problem is with the length of the “client hello” being greater than one byte, as mentioned in the openssl ticket above.

(For clarity, I understand this to be the length of the length, that is the amount of data required to represent the number of bytest in the hello … so if the hello is longer than 255 bytes then this bug in the server will not be found … and the length of the hello may be reduced by limiting the ciphersuites chosen)

[pixilla@…]

Replying to william@…:

Note that they also fail if I use the +gnutls variant of curl

curl +gnutls fails for me as well.

If I force TLSv1 (which curl allows me to do) my fetches work OK. I expect this reduces the length of the “client hello” and that allows things to work.

Forcing tlsv1 does not work for me. In the ticket above this patch ​cn22286 is referenced to fix a bug apparently introduced in 1.0.1. With this patch curl tlsv1 still fails in my tests.

Here is an mit.edu fail. I feel this problem is wide spread.

curl -tlsv1 https://owa.mit.edu/

[pixilla@…]

Replying to pixilla@…:

curl -tlsv1 ​https://owa.mit.edu/

The above tlsv1 flag and url are flawed.

Adding --ciphers works:

curl --tlsv1 --ciphers '3DES' https://owa.exchange.mit.edu/owa/auth/logon.aspx

This works for me.

[alex@…]

The reason of the issue is that new openssl doesn't automatically detect SSL version. In your example you can explicitly specify that you wan't to use SSL3: $ curl -14k --sslv3 ​https://api-aa-3t.paypal.com/2.0/ Replying to ryandesign@…:

I'm not familiar with the option "-14k" but I get:

$ curl -14k https://api-aa-3t.paypal.com/2.0
curl: (35) Unknown SSL protocol error in connection to api-aa-3t.paypal.com:443

Or trying verbosely without that option:

$ time curl -v https://api-aa-3t.paypal.com/2.0
* About to connect() to api-aa-3t.paypal.com port 443 (#0)
*   Trying 66.211.168.195...
* connected
* Connected to api-aa-3t.paypal.com (66.211.168.195) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to api-aa-3t.paypal.com:443 
* Closing connection #0
curl: (35) Unknown SSL protocol error in connection to api-aa-3t.paypal.com:443 

real	1m0.175s
user	0m0.048s
sys	0m0.021s

Is that what you're seeing as well? If so, I doubt it's a MacPorts-specific problem, since all we did was update the openssl port to version 1.0.1. Have you reported the problem to the developers of openssl / curl / paypal? (I'm not sure which is at fault.)

Trying to connect to ​https://trac.macports.org/ instead works fine.

[jpr5@…]

This problem is not specific to curl - it is specific to openssl 1.0.1. Whether it's "the server's fault" or the library's fault, the problem still persists for components that use openssl 1.0.1. The underlying issue is not resolved.

I see the rt.openssl.org bug link; what is the actual resolution here? It looks like there might be a patch, but AFAICT it hasn't been incorporated yet? I did a port selfupdate just now and openssl isn't regarded as outdated.

[ryandesign@…]

Has duplicate #34120.

[jpr5@…]

FWIW, I discovered this ticket in pursuit of a hang in a ruby SOAP client.

I was able to get things working with openssl 1.0.1 by:

• installing the openssl 1.0.1 library from source and
  • applying ​http://cvs.openssl.org/chngview?cn=22286
  • applying ​http://cvs.openssl.org/chngview?cn=22306
  • building with -DOPENSSL_NO_TLS1_2_CLIENT

• setting the OpenSSL::SSL::Context#options to OpenSSL::SSL::OP_NO_TLSv1 << 1
  • needed because the openssl layer in Ruby doesn't know about OpenSSL::SSL::OP_NO_TLSv1_2 yet
  • in my case, this was a monkeypatch to httpclient-2.1.5.2/lib/httpclient/session.rb

[jpr5@…]

OR, revert to the last known-working version (1.0.0h):

svn checkout -r 90715 http://svn.macports.org/repository/macports/trunk/dports/devel/openssl
cd openssl
sudo port install

From current openssl dev commentary, I don't get a sense that there's any strong motivation to "make it work like it used to" (​http://cvs.openssl.org/timeline, comments WRT PR#2771). And there doesn't seem to be any system-wide, run-time way to disable TLS v1.2 - patches + recompile seems to be the only current option.

Given what looks like a large number of popular websites that are affected by this, and the growing number of duplicate tickets popping up related to this library, I'd like to register my vote for macports to revert back to rev 90715 (openssl 1.0.0h) until this gets resolved.

[jpr5@…]

Cc Me!

[pixilla@…]

Has duplicate #33944

[dluke@…]

Note that this appears to be fixed in 1.0.1a

[pixilla@…]

I opened this ticket and I believe 1.0.1a fixes all related issues so I'm closing as fixed.

[jpr5@…]

Confirmed fixed for my case (ruby).

[sawright@…]

I upgraded, restarted services and even rebooted, but am still experiencing failure.

Command:

ldapsearch -d 1 -x -L -v -H ldaps://ldap.server.address:port

Fails at TLS:

TLS trace: SSL_connect:SSLv2/v3 write client hello A
TLS trace: SSL_connect:error in SSLv2/v3 read server hello A
TLS: can't connect: .
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

[diego@…]

hm. this still fails on my box after updating openssl to 1.0.1a and recompiling curl:

$ curl -v https://graph.facebook.com/oauth/access_token
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 69.171.228.43...
* connected
* Connected to graph.facebook.com (69.171.228.43) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to graph.facebook.com:443 
* Closing connection #0
curl: (35) Unknown SSL protocol error in connection to graph.facebook.com:443

Doing it fine when forcing sslv3:

$ curl -sslv3 -v https://graph.facebook.com/oauth/access_token
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 69.171.228.43...
* connected
* Connected to graph.facebook.com (69.171.228.43) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
* 	 subject: C=US; ST=California; L=Palo Alto; O=Facebook, Inc.; CN=*.facebook.com
* 	 start date: 2010-01-13 00:00:00 GMT
* 	 expire date: 2013-04-11 23:59:59 GMT
* 	 common name: *.facebook.com (matched)
* 	 issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3
* 	 SSL certificate verify ok.
> GET /oauth/access_token HTTP/1.1
> User-Agent: curl/7.25.0 (x86_64-apple-darwin10.8.0) libcurl/7.25.0 OpenSSL/1.0.1a zlib/1.2.6 libidn/1.22
> Host: graph.facebook.com
> Accept: */*
> 
< HTTP/1.1 400 Bad Request
< Access-Control-Allow-Origin: *
< Cache-Control: no-store
< Content-Type: text/javascript; charset=UTF-8
< Expires: Sat, 01 Jan 2000 00:00:00 GMT
< Pragma: no-cache
< WWW-Authenticate: OAuth "Facebook Platform" "invalid_request" "Missing redirect_uri parameter."
< X-FB-Rev: 546753
< X-FB-Debug: eKKrranEotqwCWaG3QbLUuuI7qF8EkxzbVW2P2yQ6po=
< X-Cnection: close
< Date: Wed, 25 Apr 2012 07:38:02 GMT
< Content-Length: 90
< 
* Connection #0 to host graph.facebook.com left intact
{"error":{"message":"Missing redirect_uri parameter.","type":"OAuthException","code":191}}* Closing connection #0

[dluke@…]

Replying to diego@…:

$ curl -v https://graph.facebook.com/oauth/access_token
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 69.171.228.43...

FYI, I can't reproduce your issue on either 10.5 or 10.7 with openssl 1.0.1a (connecting both to that url and to the IP address, as I 'normally' get a different one when resolving that hostname).

[pixilla@…]

Replying to diego@…:

$ curl -v https://graph.facebook.com/oauth/access_token
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 69.171.228.43...

I can reproduce this:

$ sw_vers -productVersion; \
  port -q installed \( name:^openssl$ or name:^curl$ \) and active; \
  curl -m5 https://69.171.228.43/oauth/access_token; \
  curl -3 -m5 https://69.171.228.43/oauth/access_token
10.5.8
  curl @7.25.0_0+ssl (active)
  openssl @1.0.1a_0 (active)
curl: (28) SSL connection timeout
curl: (51) SSL: certificate subject name '*.facebook.com' does not match target host name '69.171.228.43'

$ sw_vers -productVersion; \
  port -q installed \( name:^openssl$ or name:^curl$ \) and active; \
  curl -m5 https://69.171.228.43/oauth/access_token; \
  curl -3 -m5 https://69.171.228.43/oauth/access_token
10.6.8
  curl @7.25.0_0+ssl+universal (active)
  openssl @1.0.1a_1+universal (active)
curl: (28) SSL connection timeout
curl: (51) SSL: certificate subject name '*.facebook.com' does not match target host name '69.171.228.43'

$ sw_vers -productVersion; \
  port -q installed \( name:^openssl$ or name:^curl$ \) and active; \
  curl -m5 https://69.171.228.43/oauth/access_token; \
  curl -3 -m5 https://69.171.228.43/oauth/access_token
10.7.2
  curl @7.25.0_0+ssl+universal (active)
  openssl @1.0.1a_1+universal (active)
curl: (28) SSL connection timeout
curl: (51) SSL: certificate subject name '*.facebook.com' does not match target host name '69.171.228.43'

Much brokeness remains for openssl 1.0.x dependent packages. Apparently, openssl has options to specify what protocols you _DO NOT_ want to offer but not what protocols you _DO_ want to offer. So whenever openssl adds new protocols developers of dependent packages must update their software to allow turning the new protocol off.

Example: Hotmail has problems with the new TLSv1.2 ​postfix.user ​postfix-openssl-workaround released

[dluke@…]

Replying to pixilla@…:

Replying to diego@…:

$ curl -v https://graph.facebook.com/oauth/access_token
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 69.171.228.43...

I can reproduce this:

no, you can't, your 'reproduction' doesn't appear to show an actual problem

you need to see

curl: (35) Unknown SSL protocol error in connection to graph.facebook.com:443

to have reproduced the issue

[pixilla@…]

Replying to dluke@…:

Replying to pixilla@…:

Replying to diego@…:

$ curl -v https://graph.facebook.com/oauth/access_token
* About to connect() to graph.facebook.com port 443 (#0)
*   Trying 69.171.228.43...

I can reproduce this:

no, you can't, your 'reproduction' doesn't appear to show an actual problem

you need to see

curl: (35) Unknown SSL protocol error in connection to graph.facebook.com:443

to have reproduced the issue

I added the "-m5" 5 second limit because that is plenty of time if it's going to work. Without "-m5" all the examples without "-3" produce this error:

$ curl https://69.171.228.43/oauth/access_token
curl: (35) Unknown SSL protocol error in connection to 69.171.228.43:443 

[pixilla@…]

$ sw_vers -productVersion; \
>   port -q installed \( name:^openssl$ or name:^curl$ \) and active; \
>   curl https://69.171.228.43/oauth/access_token;
10.5.8
  curl @7.25.0_0+ssl (active)
  openssl @1.0.1a_0 (active)
curl: (35) Unknown SSL protocol error in connection to 69.171.228.43:443 

$ sw_vers -productVersion; \
>   port -q installed \( name:^openssl$ or name:^curl$ \) and active; \
>   curl https://69.171.228.43/oauth/access_token;
10.6.8
  curl @7.25.0_0+ssl (active)
  openssl @1.0.1a_0 (active)
curl: (35) Unknown SSL protocol error in connection to 69.171.228.43:443 

$ sw_vers -productVersion; \
>   port -q installed \( name:^openssl$ or name:^curl$ \) and active; \
>   curl https://69.171.228.43/oauth/access_token;  
10.7.2
  curl @7.25.0_0+ssl (active)
  openssl @1.0.1a_0 (active)
curl: (35) Unknown SSL protocol error in connection to 69.171.228.43:443 

[dluke@…]

Replying to pixilla@…:

I added the "-m5" 5 second limit because that is plenty of time if it's going to work. Without "-m5" all the examples without "-3" produce this error:

$ curl https://69.171.228.43/oauth/access_token
curl: (35) Unknown SSL protocol error in connection to 69.171.228.43:443 

ack. This should probably be reported upstream, then.... If I were the maintainer, I would probably close this ticket.

[pixilla@…]

diego: Please find a curl options combination that works with openssl 1.0.1a and your target.

$ curl -3 https://69.171.228.43/oauth/access_token
curl: (51) SSL: certificate subject name '*.facebook.com' does not match target host name '69.171.228.43'
$ curl --ciphers '3DES' https://69.171.228.43/oauth/access_token
curl: (51) SSL: certificate subject name '*.facebook.com' does not match target host name '69.171.228.43'