Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#62468 closed defect (fixed)

cyrus-sasl2 @2.1.27_2+kerberos: _sasl_locate_entry() in Cyrus SASL's lib/dlopen.c assumes a leading underscore is needed for calls to dlsym()

Reported by: steven-michaud (Steven Michaud) Owned by: ryandesign (Ryan Carsten Schmidt)
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc:
Port: cyrus-sasl2

Description (last modified by kencu (Ken))

I've installed cyrus-sasl2 @2.1.27_2+kerberos on macOS 11.2.3 and have been using it with postfix @3.5.9_0+dovecot_sasl+pcre+sasl+tls and dovecot @2.3.11.3_2. Sending mail with STARTTLS and the SASL PLAIN mechanism simply didn't work. I've spent the last several days debugging the problem, and now I've found the cause.

Here's a fragment of code from _sasl_locate_entry():

#if defined(DLSYM_NEEDS_UNDERSCORE) || (defined(__OpenBSD__) && !defined(__ELF__))
    snprintf(adj_entryname, sizeof adj_entryname, "_%s", entryname);
#endif

    *entry_point = NULL;
    *entry_point = dlsym(library, adj_entryname);
    if (*entry_point == NULL) {
#if 0 /* This message appears to confuse people */
      _sasl_log(NULL, SASL_LOG_DEBUG,
                "unable to get entry point %s: %s", adj_entryname,
                dlerror());
#endif
      return SASL_FAIL;
    }

By means of various tests, and by looking at libsasl2.3.dylib in a disassembler, I've found that (in your distro) this code is behaving as if DLSYM_NEEDS_UNDERSCORE is defined. Of course it shouldn't be. I don't know why. The sasl_cv_dlsym_adds_uscore test in configure.ac looks like it should work. Could your cyrus_sasl2 distro have been compiled on something other than a Mac? In other words, might it have been cross-compiled on some other kind of system (e.g. Linux)?

Every time I try to send mail via STARTTLS and the SASL PLAIN mechanism, I get the error message "No worthy mechs found". Postfix is correctly set up -- the same setup works fine on other machines which use Cyrus SASL from other providers (like Apple). The reason is that my connection's list of mechanisms (sasl_client_conn_t.mech_list) only includes the "EXTERNAL" mechanism. sasl_client_add_plugin() is only ever called on this mechanism. The reason it isn't called on all the others (including the PLAIN mechanism) is that _sasl_locate_entry() (called from _sasl_plugin_load()) always fails, because it always adds an underscore in front of entryname.

Attachments (2)

ticket62468-hook-library-diff.txt (28.5 KB) - added by steven-michaud (Steven Michaud) 3 years ago.
HookCase hook library I tested with
smtp.sh (102 bytes) - added by steven-michaud (Steven Michaud) 3 years ago.
'smtp' launcher that loads my hook library

Download all attachments as: .zip

Change History (40)

Changed 3 years ago by steven-michaud (Steven Michaud)

HookCase hook library I tested with

comment:1 Changed 3 years ago by steven-michaud (Steven Michaud)

I just added the source code for a HookCase hook library I've been testing with, as a patch on https://github.com/steven-michaud/HookCase/blob/master/HookLibraryTemplate/hook.mm.

https://github.com/steven-michaud/HookCase

comment:2 Changed 3 years ago by steven-michaud (Steven Michaud)

Port: cyrus-sasl2 @2.1.27_2+kerberos added

Changed 3 years ago by steven-michaud (Steven Michaud)

Attachment: smtp.sh added

'smtp' launcher that loads my hook library

comment:3 Changed 3 years ago by kencu (Ken)

it checks here (configure.ac)

dnl Do we need leading underscores on our symbols?

AC_CHECK_PROGS(NM, nm)

AC_MSG_CHECKING(for underscore before symbols)
AC_CACHE_VAL(sasl_cv_uscore,[
    echo "int main(){int i=1; return 0;}
    void foo(){int i=6;}" > conftest.c
    ${CC} -o a.out conftest.c > /dev/null
    if (${NM} a.out | grep _foo) > /dev/null; then
      sasl_cv_uscore=yes
    else
      sasl_cv_uscore=no
    fi])
AC_MSG_RESULT($sasl_cv_uscore)
rm -f conftest.c a.out

and on my Mojave system, finds this:

configure:15311: checking for underscore before symbols
configure:15327: result: yes

comment:4 Changed 3 years ago by kencu (Ken)

but in the next section, comes something interesting that could be the issue. What system are you on? On some systems, this implicit-function-declaration is raised to an error.

configure:15333: checking whether dlsym adds the underscore for us
configure:15355: /usr/bin/clang -o conftest -Wall -W -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -arch x86_64 -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -arch x86_64 conftest.c  -ldl >&5
conftest.c:37:18: warning: unused variable 'i' [-Wunused-variable]
void foo() { int i=0;}
                 ^
conftest.c:40:23: warning: implicitly declaring library function 'exit' with type 'void (int) __attribute__((noreturn))' [-Wimplicit-function-declaration]
    if(ptr1 && !ptr2) exit(0); } exit(1); }
                      ^
conftest.c:40:23: note: include the header <stdlib.h> or explicitly provide a declaration for 'exit'
2 warnings generated.
configure:15355: $? = 0
configure:15355: ./conftest
configure:15355: $? = 0
configure:15370: result: yes
Last edited 3 years ago by kencu (Ken) (previous) (diff)

comment:5 Changed 3 years ago by steven-michaud (Steven Michaud)

I used 'smtp.sh' (above) to load my hook library. I replaced 'smtp' in Postfix's master.cf with 'smtp.sh'.

comment:6 Changed 3 years ago by steven-michaud (Steven Michaud)

I'm on macOS 11.2.3. I used "port install" for everything (including cyrus-sasl2 @2.1.27_2+kerberos). I didn't do any explicit compiling.

comment:7 Changed 3 years ago by kencu (Ken)

Yeah, it probably fails that dlsym adds the underscore for us test because of Wimplicit-function-declaration being raised to an error.

can you check in your config.log?

Last edited 3 years ago by kencu (Ken) (previous) (diff)

comment:8 Changed 3 years ago by kencu (Ken)

cd `port work cyrus-sasl2`
cd cy*
bbedit config.log

and search for:

dlsym adds the underscore for us

comment:9 Changed 3 years ago by steven-michaud (Steven Michaud)

can you check in your config.log?

Where is is? I couldn't find it anywhere under /opt/local.

comment:10 Changed 3 years ago by kencu (Ken)

I figured that might happen, so I just outlined how to find it one box above :>

comment:11 Changed 3 years ago by steven-michaud (Steven Michaud)

Oh well. In any case I still can't find it. port work cyrus-sasl2 resolves to my home directory, and there's nothing there that I didn't explicitly create myself.

comment:12 Changed 3 years ago by kencu (Ken)

really? here it is then

$ port work cyrus-sasl2
/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_security_cyrus-sasl2/cyrus-sasl2/work

comment:13 Changed 3 years ago by kencu (Ken)

Oh -- you have to have the work directory for it to be found, of course. Do this:

sudo port -v configure cyrus-sasl2
cd `port work cyrus-sasl2`
cd cy*
bbedit config.log

or just wait if you like, and someone will do it in time. It's about 99.9999% likely that is going to be the issue.

We've seen hundreds of these since Apple raised Wimplicit-function-declaration to an error in Xcode 12.

comment:14 Changed 3 years ago by steven-michaud (Steven Michaud)

"port work cyrus-sasl2" produces no output. And there are no "work" directories at all under /opt/local. I used "sudo find /opt/local -name work".

comment:15 Changed 3 years ago by kencu (Ken)

see one box above.

comment:16 Changed 3 years ago by kencu (Ken)

Description: modified (diff)

comment:17 Changed 3 years ago by steven-michaud (Steven Michaud)

Yup. Now I see exactly what you reported in comment 4.

comment:18 Changed 3 years ago by kencu (Ken)

You probably don't. You probably see this:

configure:15333: checking whether dlsym adds the underscore for us
configure:15355: /usr/bin/clang -o conftest -Wall -W -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -arch x86_64 -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -arch x86_64 conftest.c  -ldl >&5
conftest.c:37:18: warning: unused variable 'i' [-Wunused-variable]
void foo() { int i=0;}
                 ^
conftest.c:40:23: error: implicitly declaring library function 'exit' with type 'void (int) __attribute__((noreturn))' [-Werror,-Wimplicit-function-declaration]
    if(ptr1 && !ptr2) exit(0); } exit(1); }
                      ^
conftest.c:40:23: note: include the header <stdlib.h> or explicitly provide a declaration for 'exit'
1 warning and 1 error generated.
configure:15355: $? = 1
configure: program exited with status 1

which is what I see on BigSur, and that exit status 1 is what gives the (incorrect) failure that leads to the issue, by defining this:

#define DLSYM_NEEDS_UNDERSCORE /**/

because it (falsely) thinks it is needed.

The fix is to do exactly what it says to do:

include the header <stdlib.h> or explicitly provide a declaration for 'exit'
Last edited 3 years ago by kencu (Ken) (previous) (diff)

comment:19 Changed 3 years ago by kencu (Ken)

There are also a bunch of these:

error: implicit declaration of function 'db_create' is invalid in C99 [-Werror,-Wimplicit-function-declaration]

that may, or may not, be another issue that needs fixing.

Good for you to notice what was wrong and to steer it in the right direction. This is an upstream issue to fix, of course, as every system running Xcode 12 on Earth is broken right now (nothing to do with MacPorts). But we'll see if we can get it fixed for you.

comment:20 Changed 3 years ago by kencu (Ken)

It looks like homebrew has the same version, and no patches, so theirs is likely just as broken as you found ours to be.

<https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/cyrus-sasl.rb>

So we'll have to make up our own patches, it seems.

comment:21 Changed 3 years ago by steven-michaud (Steven Michaud)

Oops, sorry. What you say in comment 18 is correct.

But I don't see the error from comment 19 at all. Edit: This is wrong. I wasn't searching from the top of the file :-(

Is there any way to just do my own patch and rebuild your distro? Please point me to documentation that tells me how. (Needless to say, I can deal with the source code. I just don't know how to get from there to something that will fit with the rest of your distros.)

Last edited 3 years ago by steven-michaud (Steven Michaud) (previous) (diff)

comment:22 Changed 3 years ago by steven-michaud (Steven Michaud)

implicit-function-declaration is raised to an error

For the time being, I'm strongly tempted to just turn this off in XCode 12. There must be a way.

comment:23 Changed 3 years ago by kencu (Ken)

for a quick fix, use a macports compiler if you're on Intel. This should do it:

sudo port clean cyrus-sasl2
sudo port install clang-11
sudo port -v install cyrus-sasl2 configure.compiler=macports-clang-11

the macports-clang compilers don't have that Werror turned on (but this needs to be turned on for arm64 to work).

What we will do is patch configure to add this:

#include <stdlib.h> 

at the right spot to make the test pass, as it should.

comment:24 Changed 3 years ago by kencu (Ken)

Here's how we generate the patch, by the way:

sudo port -v extract cyrus-sasl2
cd `port work cyrus-sasl2`
cd cy*
sudo cp configure configure.orig
bbedit configure

add this in the right place:

#include <stdlib.h> 

then

diff -u configure.orig configure > ~/patch-cyrus-fix-implicit-dlysm.diff

then

cd ~
sudo port -v install cyrus-sasl2

(builds with your patched configure, hopefully fixed up as it should be).

and then if you get that far and are motivated, I can show you how to make a PR and fix this for everyone.

If you are not that motivated, attach your patch here to this ticket instead.

If you really like this port, and you (obviously) know what you're doing, volunteer to be the worldwide port maintainer, and make sure it always works correctly for everyone!

comment:25 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Owner: set to ryandesign
Port: @2.1.27_2+kerberos removed
Status: newaccepted
Summary: _sasl_locate_entry() in Cyrus SASL's lib/dlopen.c assumes a leading underscore is needed for calls to dlsym()cyrus-sasl2 @2.1.27_2+kerberos: _sasl_locate_entry() in Cyrus SASL's lib/dlopen.c assumes a leading underscore is needed for calls to dlsym()

comment:26 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Resolution: fixed
Status: acceptedclosed

In ba36fd455188921e2bf22d73e23a2e91f36f332f/macports-ports (master):

cyrus-sasl2: Fix implicit declaration of function

Closes: #62468

comment:27 Changed 3 years ago by steven-michaud (Steven Michaud)

So can I upgrade now to cyrus-sasl2 @2.1.27_3? I tried "sudo port upgrade outdated" and "sudo port -d sync". Neither of them did the trick.

comment:28 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Wait at least 1 hour after the commit for it to propagate to the public rsync server.

comment:29 Changed 3 years ago by kencu (Ken)

So Steven, I see someone did the patch for you, but in that "teach a man to fish" way I think you were coming along nicely.

MacPorts needs more maintainers, especially when people actually use the software and know when it's not working right.

So if this has given you any enthusiasm dive in a bit, and if you get there, please feel free volunteer to help us maintain this and other software you know about. You can ask any questions you may have on the developer mailing list.

comment:30 Changed 3 years ago by steven-michaud (Steven Michaud)

I still can't upgrade cyrus-sasl2 @2.1.27_2+kerberos. Even the command "sudo port upgrade cyrus-sasl2" didn't work.

And in any case my service provider seems to have blacklisted my test server :-( Once I found out how to use my hook library to work around the bug, something like 25 test messages (to myself) got delivered from my mail queue almost simultaneously. That seems to have been what triggered the blacklisting. Now I get a different error ("Relay access denied (in reply to RCPT TO command))"), before the SASL client even gets initialized.

I'll try to persuade them to undo the blacklisting. Then, once I'm able to upgrade to cyrus-sasl2 @2.1.27_3+kerberos, I'll be able to test it.

comment:31 Changed 3 years ago by steven-michaud (Steven Michaud)

Thanks for the offer to make me a port maintainer, but I'm going to decline it. I'm retired :-) I don't want to shoulder that kind of burden. However, I'm happy to report any more problems I find, and to fix them if possible.

My current interest in MacPorts is to work up a replacement for Apple's late and lamented macOS/OS X Server. It actually still exists, but only as a pale ghost of what it used to be. As of macOS 10.14 (Mojave) it no longer supports the services that most people want -- things like mail, web service, dns and dhcp. I've run the "original" OS X Server for many years, but now it's getting very long in the tooth, and doesn't run on current versions of macOS. I figured that if I based my replacement on MacPorts, it'd be much easier to keep it up to date.

This replacement is just for my own use. But if it goes well I might create a Github project to describe how I implemented it.

Last edited 3 years ago by steven-michaud (Steven Michaud) (previous) (diff)

comment:32 Changed 3 years ago by steven-michaud (Steven Michaud)

Turns out what I called blacklisting was actually my fault. I inadvertently changed the Postfix "smtp_sasl_auth_enable" setting from "yes" to "no". When I changed it back my hook library workaround started working correctly again.

But I'm still not able to upgrade to cyrus-sasl2 @2.1.27_3+kerberos. So I can't test it.

comment:33 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

You have run sudo port selfupdate to receive the fix?

Note that there is already some work in MacPorts to provide replacements for what Apple provided in OS X Server, such as the calendar-contacts-server, clamav-server, dns-server, macos-vpn-server, and mail-server ports.

comment:34 Changed 3 years ago by steven-michaud (Steven Michaud)

"sudo port selfupdate" made no difference, as you'd expect since it only effects "base sources". But I tried "sudo port upgrade outdated" again, as it recommended, and this time it did upgrade cyrus-sasl2 to 2.1.27_3. I guess it takes about 24 hours for new updates to migrate out fully.

I'll test it once I've had my dinner :-)

comment:35 in reply to:  34 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to steven-michaud:

"sudo port selfupdate" made no difference, as you'd expect since it only effects "base sources".

It does not. selfupdate updates base, if needed, and always updates your ports tree, unless you use the --no-sync flag.

But I tried "sudo port upgrade outdated" again, as it recommended, and this time it did upgrade cyrus-sasl2 to 2.1.27_3. I guess it takes about 24 hours for new updates to migrate out fully.

It takes about 1 hour for commits to move from our repository to the public rsync server, after which time you can run sudo port selfupdate or sudo port sync to receive those changes, and then do whatever install or upgrade you were trying to do.

comment:36 Changed 3 years ago by steven-michaud (Steven Michaud)

Here's the output I got from "sudo port selfupdate":

--->  Updating MacPorts base sources using rsync
MacPorts base version 2.6.4 installed,
MacPorts base version 2.6.4 downloaded.
--->  Updating the ports tree
--->  MacPorts base is already the latest version

The ports tree has been updated. To upgrade your installed ports, you should run
  port upgrade outdated

So it updates both the "base sources" and the "ports tree". But that isn't enough by itself.

It seems you need to do both "sudo port selfupdate" and "sudo port upgrade outdated". And if I'd done both about an hour after you landed your fix, my copy of cyrus-sasl2 would have been upgraded to 2.1.27_3. I'll remember that for future reference.

Last edited 3 years ago by steven-michaud (Steven Michaud) (previous) (diff)

comment:37 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Updating the ports tree by itself using sudo port sync is enough. sudo port selfupdate does nothing more than that, unless a new version of MacPorts base had been released, but it hasn't.

comment:38 Changed 3 years ago by steven-michaud (Steven Michaud)

I just redid my test of sending email to myself via STARTTLS and SASL PLAIN authentication to my service provider. It worked fine.

Thank you for fixing this so quickly!

I'll take a look at the ports you mentioned in comment 33, particularly "mail-server". I didn't know about them.

Last edited 3 years ago by steven-michaud (Steven Michaud) (previous) (diff)
Note: See TracTickets for help on using tickets.