Opened 3 years ago

Closed 3 years ago

Last modified 3 months ago

#59772 closed defect (fixed)

python38: fails on Leopard: Undefined symbols: :info:build "_pthread_threadid_np"

Reported by: kencu (Ken) Owned by: jmroot (Joshua Root)
Priority: Normal Milestone:
Component: ports Version:
Keywords: leopard tiger Cc: kencu (Ken), udbraumann, a-chaudhari
Port: python38

Description (last modified by kencu (Ken))

:info:build Undefined symbols:
:info:build   "_pthread_threadid_np", referenced from:
:info:build       _PyThread_get_thread_native_id in libpython3.8.a(thread.o)

that symbol exists on SnowLeopard and later. In llvm-9.0, I handled a similar issue like this:

diff --git llvm_master/lib/Support/Unix/Threading.inc macports_master/lib/Support/Unix/Threading.inc
index ed9a9656305..e8f9a13860f 100644
--- llvm_master/lib/Support/Unix/Threading.inc
+++ macports_master/lib/Support/Unix/Threading.inc
@@ -14,6 +14,7 @@
 #include "llvm/ADT/Twine.h"
 
 #if defined(__APPLE__)
+#include <Availability.h>
 #include <mach/mach_init.h>
 #include <mach/mach_port.h>
 #endif
@@ -153,8 +154,10 @@ void llvm::set_thread_name(const Twine &Name) {
   ::pthread_setname_np(::pthread_self(), "%s",
     const_cast<char *>(NameStr.data()));
 #elif defined(__APPLE__)
+#if __MAC_OS_X_VERSION_MAX_ALLOWED >= 1060
   ::pthread_setname_np(NameStr.data());
 #endif
+#endif
 }
 
 void llvm::get_thread_name(SmallVectorImpl<char> &Name) {
-- 

Attachments (2)

python38-fail-leopard.log (117.3 KB) - added by kencu (Ken) 3 years ago.
patch-python38-leopard-tiger.diff (2.1 KB) - added by kencu (Ken) 3 years ago.

Download all attachments as: .zip

Change History (28)

Changed 3 years ago by kencu (Ken)

Attachment: python38-fail-leopard.log added

comment:1 Changed 3 years ago by kencu (Ken)

Description: modified (diff)

Changed 3 years ago by kencu (Ken)

comment:2 Changed 3 years ago by jmroot (Joshua Root)

Is there a reason they're not just using pthread_self() in the first place?

comment:3 Changed 3 years ago by kencu (Ken)

I think that's the special insight you always bring to these questions, Josh.

comment:4 Changed 3 years ago by jmroot (Joshua Root)

Probably because it returns a pointer to a struct and not a "real" thread id number. There may be concerns about that not being guaranteed to be unique over the lifetime of the process. I don't know if using an undocumented syscall will give better results though. Have you tested that it returns sane values? At minimum, I'd want to start a few threads and check that the ids it returns for them are all different, and maybe kill one and spawn a new one and check again.

comment:5 Changed 3 years ago by kencu (Ken)

I copied exactly the exact same call block python uses now on linux, in the same file -- see the lines immediately above (in the first) and below (in the second).

comment:6 Changed 3 years ago by jmroot (Joshua Root)

Yes but there's no guarantee that it does the same thing on Darwin as it does on Linux.

comment:7 Changed 3 years ago by kencu (Ken)

(Heh -- I was typing as you were typing --- )

but -- if you show me how to do what you ask -- me or dgellieus (sp?) will do it.

I am out of commission for a week or two now, however, so if I don't answer, it's because of that.

comment:8 Changed 3 years ago by kencu (Ken)

well well well <http://elliotth.blogspot.com/2012/04/gettid-on-mac-os.html>

"If you poke around, you'll see that the Darwin/xgnu kernel actually has a gettid(2) system call. But before you get excited, you'll find that it's completely unrelated to the Linux gettid(2). It returns the "per-thread override identity", which is a uid and gid that a thread might be operating under (like a per-thread setuid(2) kind of facility). No use to us."

comment:9 Changed 3 years ago by SerpentChris (Chris Calderon)

Can Ken's patch be added to this port so that it will build? Also does anyone know how to contact the Python developers to let them know about this? I believe the problems with building Python on older Macs stem from the recent removal of their Tiger build bot.

comment:10 Changed 3 years ago by kencu (Ken)

My patch is not really right, Josh is correct. It does allow the build, but I need to fix it better. Just have one other thing to do first, then I'll work on it.

comment:11 Changed 3 years ago by kencu (Ken)

I forgot about this ticket...sorry.

comment:12 Changed 3 years ago by kencu (Ken)

Cc: kencu added

comment:13 Changed 3 years ago by kencu (Ken)

Some resources I accumulated in hopes of coming up with some kind of implementation for pthread_threadid_np on < 10.6 that might not be total garbage:

<https://stackoverflow.com/questions/1540603/mac-iphone-is-there-a-way-to-get-a-thread-identifier-without-using-objective-c>

<https://stackoverflow.com/questions/21091000/how-to-get-thread-id-of-a-pthread-in-linux-c-program>

<https://developer.apple.com/library/archive/documentation/General/Conceptual/ConcurrencyProgrammingGuide/ThreadMigration/ThreadMigration.html#//apple_ref/doc/uid/TP40008091-CH105-SW1>

<https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/KernelProgramming/scheduler/scheduler.html>

<https://robert.sesek.com/2018/7/debugging_early_program_startup_on_macos.html>

Alternatively, if none of these lead to something acceptable, it's not the end of the world to keep a local Tiger patch here hidden away in MacPorts; these threadid's are mostly used for debugging anyway, and goodness help anyone who is trying to do some serious debugging on Tiger these days...people just want python38 to build so they're not left high and dry...

Last edited 3 years ago by kencu (Ken) (previous) (diff)

comment:14 Changed 3 years ago by kencu (Ken)

This snippet is getting very close, just need to flesh out the details of replicating the full functionality of pthread_threadid_np. This might need to go into legacysupport as it is a tad complicated, but perhaps the python source file will accept these headers being added ...

#include <unistd.h>

#include <errno.h>
#include <iostream>
#include <pthread.h>
#include <sys/syscall.h>

#include <mach/kern_return.h>
#include <mach/thread_info.h>
#include <mach/thread_act.h>

uint64_t my_pthread_threadid_np(pthread_t thread) {
    mach_port_name_t port=pthread_mach_thread_np(thread);

    thread_identifier_info_data_t info;
    mach_msg_type_number_t info_count=THREAD_IDENTIFIER_INFO_COUNT;
    kern_return_t kr=thread_info(port,
                                 THREAD_IDENTIFIER_INFO,
                                 (thread_info_t)&info,
                                 &info_count);
    if(kr!=KERN_SUCCESS) {
        /* you can get a description of the error by calling
         * mach_error_string(kr)
         */
        return 0;
    } else {
        return info.thread_id;
    }
}

int main() {
 std::cout << "getpid()=" << getpid() << std::endl;

 std::cout << "pthread_self()=" << pthread_self() << std::endl;
 uint64_t tid;
 pthread_threadid_np(NULL, &tid);
 std::cout << "pthread_threadid_np()=" << tid << std::endl;

 std::cout << "syscall(SYS_thread_selfid)=" << syscall(SYS_thread_selfid) << std::endl;
 
//for (int i=0; i < 100; i++) { 
 mach_port_t newtid = pthread_mach_thread_np(pthread_self());
 std::cout << "pthread_mach_thread_np()=" << (uint64_t)newtid << std::endl;
//}
 
 std::cout << "my_pthread_threadid_np()=" << my_pthread_threadid_np(pthread_self()) << std::endl;
 
 return 0;
}


// getting very close to replicating the function
// check the man page for pthread_threadid_np for details on implementation
// probably can find it in Apple Open Source as well

comment:15 Changed 3 years ago by kencu (Ken)

this looks like it will do it:

#include <errno.h>
#include <pthread.h>
#include <mach/kern_return.h>
#include <mach/thread_info.h>
#include <mach/thread_act.h>

int pthread_threadid_np(pthread_t thread, __uint64_t *thread_id) {

    if (thread_id == NULL) {
        return EINVAL;
    }

    if (thread == NULL) {
        thread = pthread_self();
        
        if (thread == NULL ) {
          return ESRCH;
        }
    }

    mach_port_name_t port = pthread_mach_thread_np(thread);

    thread_identifier_info_data_t info;
    mach_msg_type_number_t info_count = THREAD_IDENTIFIER_INFO_COUNT;

    kern_return_t kr = thread_info(port,
                                   THREAD_IDENTIFIER_INFO,
                                   (thread_info_t)&info,
                                   &info_count);
    if(kr != KERN_SUCCESS) {
        return ESRCH;
    } else {
        *thread_id = info.thread_id;
        return 0;
    }
}

comment:16 Changed 3 years ago by kencu (Ken)

but not on Tiger or Leopard -- grrr -- where these are actually needed.

I have surveyed the web, and prior to SnowLeopard, and even now in many cases, this is used for a thread_id:

mach_port_name_t port = pthread_mach_thread_np(thread);

I think for the purposes of a reasonable approximation, that is what we should use on Tiger and Leopard until (and if) such time as we figure out that there is actually kernel support in Tiger and Leopard for the full pthread_threadid_np that is used on SnowLeopard and later.

Last edited 3 years ago by kencu (Ken) (previous) (diff)

comment:17 Changed 3 years ago by kencu (Ken)

Resolution: fixed
Status: assignedclosed

In 17a67bbbeb98bfbc27d006d41c699d5c84a489dc/macports-ports (master):

python38: fix build on older systems

there is no copyfile on Tiger
work around lack of pthread_threadid_np on < 10.6

closes: #59827
closes: #59772

comment:18 Changed 3 years ago by kencu (Ken)

Resolution: fixed
Status: closedreopened

the patch i borrowed had a typo that I only noted on 14th reading. very sorry. will redo it once fixed up and I rerun all the tests.

Last edited 3 years ago by kencu (Ken) (previous) (diff)

comment:19 Changed 3 years ago by udbraumann

Cc: udbraumann added

comment:20 Changed 3 years ago by kencu (Ken)

OK -- four months now. I still have the fix sitting here. OK to just push it now?

comment:21 Changed 3 years ago by proxsi

I have the same issue.

comment:22 Changed 3 years ago by a-chaudhari

Cc: a-chaudhari added

comment:23 Changed 3 years ago by ken-cunningham-webuse

Resolution: fixed
Status: reopenedclosed

In 7da358db764df2913c58385f3857e248191cc460/macports-ports (master):

python38: fix build on < 10.6

work around no copyfile on < 10.5
work around no pthread_threadid_np on < 10.6

closes: #60342
closes: #59772
closes: #59827

Thanks to @dgelessus as well for valuable input
There is a comprehensive patch upstream being considered
for a future python release.

comment:24 in reply to:  14 Changed 10 months ago by barracuda156

Replying to kencu:

This snippet is getting very close, just need to flesh out the details of replicating the full functionality of pthread_threadid_np. This might need to go into legacysupport as it is a tad complicated, but perhaps the python source file will accept these headers being added ...

Has it ever made into legacysupport?

comment:25 Changed 10 months ago by kencu (Ken)

Nah, I ran out of steam for the proper fix -- but you can take over and get it done!

comment:26 in reply to:  25 Changed 3 months ago by barracuda156

Replying to kencu:

Nah, I ran out of steam for the proper fix -- but you can take over and get it done!

What do we need to implement? pthread_setname_np, pthread_threadid_np, any friends?

Yeah, it gets annoying to have to deal with these every time. Makes better sense to fix once for all.

Note: See TracTickets for help on using tickets.