gcc and gcc-devel run fails libSystem.B.dylib not found

[AlanTAMU]

dyld[93269]: dyld cache '/System/Library/dyld/dyld_shared_cache_arm64e' not loaded: syscall to map cache into shared region failed
dyld[93269]: Library not loaded: /usr/lib/libSystem.B.dylib
  Referenced from: /Users/alan/bench/lapack-test/a.out
  Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/usr/local/lib/libSystem.B.dylib' (no such file)

[ryandesign (Ryan Carsten Schmidt)]

What version of macOS? What version of Xcode? What computer? Have you searched for this error message? Here are some results I found; maybe they are relevant to you:

• ​https://developer.apple.com/forums/thread/697494
• ​https://developer.apple.com/forums/thread/669533

[chrstphrchvz (Christopher Chavez)]

MacPorts’ Trac instance does not support commenting by email (maybe it should, but that is a separate discussion). Email notifications from Trac have reply-to set to the macports-users list, which is ​where the reporter’s response has ended up:

Thank you very much for your response.

I apologize for being imprecise.

1. The computer is a 2020 MacBook Air m1 and the operating system is MacOS 12.3. However, I have the same problem on a 2013 Intel MacBook Pro running MacOS 11.6.5.

2. The code is a Fortran code that I compiled using gfortran. The code solves partial differential equations for scientific modeling.

3. The basic problem is that the code has large static arrays so the static storage exceeds 4 GBytes. It seems odd to me that a modern 64bit computer and operating system is limited to 32bit addressing for arrays.

4. I had found the URLs you listed in your email but I didn’t find them helpful. That could certainly be because I didn’t fully understand them. My limitation.

Many thanks for your reply anyway. Much appreciated.

Best regards.

Alan

[ryandesign (Ryan Carsten Schmidt)]

Then the first URL I posted is helpful in that it is from someone else having the same problem with large static arrays. That lets us know it is not a problem specific to your computer but a more general problem. However that post also says the same code works fine on Intel whereas for you it fails on Intel. I don't know what's meant by "large". Perhaps there is some value of "large" that works on Intel but not on Apple Silicon and an even larger value of "large" that works on neither system.

I doubt the problem is specific to MacPorts. You could try reporting the problem to the developers of gcc and/or to Apple.

[kencu (Ken)]

gcc on arm64 darwin is currently a project spearheaded by one lone developer, and is not yet part of official gcc releases.

He is very responsive.

I suggest you open an issue on his github gcc repo here if you want this fixed :

​https://github.com/iains/gcc-darwin-arm64/issues

Now if you are also seeing this reproducibly on an official Intel release, he may ask you to open a gcc bug on bugzilla, or may do it humself.

[fvaccari]

Me too have this problem, and couldn't solve it up to now. Even the simplest FORTRAN code asking for a large array crashes at runtime. I'm actually the "effevi" who commented on

​https://developer.apple.com/forums/thread/697494

but nobody followed up on that. It is puzzling me that I can compile and run that program on a Linux VM with just 8 GB of RAM.

The original program that fails to execute on my new M1 Mac can be compiled and run with no issues un my old Intel Mac with 16 GB RAM, but fails on a new M1 Mac with 16 GB RAM. It fails also an another M1 Mac with 64 GB RAM. But I can run it on a Linux VM with 8 GB RAM assigned, provided that I add there the option

-mcmodel=large

at compile time. Unfortunately that option leads to errors at compile time on any Mac (Intel or Apple Silicon) I've tried it on.

It's actually the single program out of a couple of hundred programs of the package I'm working on that I can't execute on the M1 Mac.

I'm adding here the link to the most detailed comment I've found on the matter

​https://developer.apple.com/forums/thread/655588?answerId=665804022#665804022

that, as far as I can understand, seems to tell that the problem occurs when non-Apple linkers are involved, but doesn't offer a solution:

"OTOH, if your third-party tools are using their own linker then you’ll have to determine whether that linker has been updated to use stub libraries. I can’t help you with that, alas."

Hopefully someone smarter than me can get some clue from the detailed description available there...

In the mean time I've experimented with with no luck using this stupid program:

program prog

complex  ::   w40(300000000)
w40=cmplx(0.0e0,0.0e0)

print *,"OK"
stop
end

that compiled with no options on the Linux VM with 8 GB of RAM can be smoothly executed:

• ndsha@gao:ndsha/Rpath gfortran prog.f90
• ndsha@gao:ndsha/Rpath ./a.out
 OK
• ndsha@gao:ndsha/Rpath 

Below a session of failed attempts at running it on the M1 Mac. I played with compile options, install_name_tool and one environment variable. Maybe someone can come out with more ideas on this while I go on with my experiments...

Sure I'm going to contact that lone developer cited by Ken while I was typing my comment...

Ciao

Franco

A plain compile:

• fv@vm1:Test/Rpath ls
README   prog.f90

• fv@vm1:Test/Rpath gfortran-mp-devel prog.f90   

• fv@vm1:Test/Rpath otool -L a.out
a.out:
	@rpath/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
	@rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

• fv@vm1:Test/Rpath ./a.out
dyld[88396]: dyld cache '/System/Library/dyld/dyld_shared_cache_arm64e' not loaded: syscall to map cache into shared region failed
dyld[88396]: Library not loaded: /usr/lib/libSystem.B.dylib
  Referenced from: /Volumes/xHD/ndsha/Shared/Test/Rpath/a.out
  Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/usr/local/lib/libSystem.B.dylib' (no such file)
[1]    88396 abort      ./a.out

With separate linking phase:

• fv@vm1:Test/Rpath gfortran-mp-devel -c prog.f90

• fv@vm1:Test/Rpath gfortran-mp-devel  prog.o -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib

• fv@vm1:Test/Rpath otool -L a.out
a.out:
	@rpath/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
	@rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

• fv@vm1:Test/Rpath ./a.out
dyld[88503]: dyld cache '/System/Library/dyld/dyld_shared_cache_arm64e' not loaded: syscall to map cache into shared region failed
dyld[88503]: Library not loaded: /usr/lib/libSystem.B.dylib
  Referenced from: /Volumes/xHD/ndsha/Shared/Test/Rpath/a.out
  Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/usr/local/lib/libSystem.B.dylib' (no such file)

[1]    88503 abort      ./a.out

Using install_name_tool, showing the path searched for the missing library:

• fv@vm1:Test/Rpath install_name_tool -change /usr/lib/libSystem.B.dylib @rpath/libSystem.B.dylib a.out
• fv@vm1:Test/Rpath otool -L a.out
a.out:
	@rpath/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
	@rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

• fv@vm1:Test/Rpath ./a.out 
dyld[88527]: dyld cache '/System/Library/dyld/dyld_shared_cache_arm64e' not loaded: syscall to map cache into shared region failed
dyld[88527]: Library not loaded: @rpath/libSystem.B.dylib
  Referenced from: /Volumes/xHD/ndsha/Shared/Test/Rpath/a.out
  Reason: tried: '/Volumes/xHD/ndsha/Shared/Test/Rpath/libSystem.B.dylib' (no such file), '/opt/local/lib/gcc-devel/gcc/aarch64-apple-darwin21/12.0.1/libSystem.B.dylib' (no such file), '/opt/local/lib/gcc-devel/gcc/aarch64-apple-darwin21/12.0.1/../../../libSystem.B.dylib' (no such file), '/Volumes/xHD/ndsha/Shared/Test/Rpath/libSystem.B.dylib' (no such file), '/opt/local/lib/gcc-devel/gcc/aarch64-apple-darwin21/12.0.1/libSystem.B.dylib' (no such file), '/opt/local/lib/gcc-devel/gcc/aarch64-apple-darwin21/12.0.1/../../../libSystem.B.dylib' (no such file), '/usr/local/lib/libSystem.B.dylib' (no such file), '/usr/lib/libSystem.B.dylib' (no such file)
[1]    88527 abort      ./a.out

Setting a path-related variable, and listing the libSystem.B.dylib related files found there:

• fv@vm1:Test/Rpath export DYLD_FALLBACK_LIBRARY_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib

• fv@vm1:Test/Rpath ll /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib | grep libSystem\.B
-rw-r--r--    1 root  wheel   303K Nov 17 06:30 libSystem.B.tbd
-rw-r--r--    1 root  wheel   303K Nov 17 06:30 libSystem.B_asan.tbd
lrwxr-xr-x    1 root  wheel    15B Feb  8 01:25 libSystem.tbd -> libSystem.B.tbd
lrwxr-xr-x    1 root  wheel    20B Feb  8 01:25 libSystem_asan.tbd -> libSystem.B_asan.tbd

• fv@vm1:Test/Rpath ./a.out
dyld[88604]: dyld cache '/System/Library/dyld/dyld_shared_cache_arm64e' not loaded: syscall to map cache into shared region failed
dyld[88604]: Library not loaded: @rpath/libSystem.B.dylib
  Referenced from: /Volumes/xHD/ndsha/Shared/Test/Rpath/a.out
  Reason: tried: '/Volumes/xHD/ndsha/Shared/Test/Rpath/libSystem.B.dylib' (no such file), '/opt/local/lib/gcc-devel/gcc/aarch64-apple-darwin21/12.0.1/libSystem.B.dylib' (no such file), '/opt/local/lib/gcc-devel/gcc/aarch64-apple-darwin21/12.0.1/../../../libSystem.B.dylib' (no such file), '/Volumes/xHD/ndsha/Shared/Test/Rpath/libSystem.B.dylib' (no such file), '/opt/local/lib/gcc-devel/gcc/aarch64-apple-darwin21/12.0.1/libSystem.B.dylib' (no such file), '/opt/local/lib/gcc-devel/gcc/aarch64-apple-darwin21/12.0.1/../../../libSystem.B.dylib' (no such file), '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libSystem.B.dylib' (no such file)
[1]    88604 abort      ./a.out

[kencu (Ken)]

we are using a current Apple linker and no other (by default at least) on MacPorts.

My guess is that you’re running out of memory addressing space for dyld to use when it tries to load in the shared libraries, because of your large array, but Iain is the only one around who might actually know for sure and sort you out.

I saw your issue on his github… good luck!

[kencu (Ken)]

OK, so the short answer here is that at present, you can't load a static data structure on the stack of that size, as dyld is unable to do the needed relocations thereafter, and when it fails it generates a false error about the library not being found.

The real error is that the library can't be loaded and linked as you're out of addressing space.

This is an Apple/ld64/dyld thing, not related to gcc or fortran or macports or ... any such thing, and whether it will ever be fixed depends.

If someone wanted to open an Apple Radar about it that might (or might not) be useful, but there is nothing for us to do here.

[kencu (Ken)]

Now I'm talking here about that fortan example -- whether that is the same exact thing that is happening in the OP's lapack-test/a.out I can't say for certain.

Seems tempting to think it might be similar, but that would require someone digging in to lapack-test/a.out and seeing what it is trying to do, exactly.

[Dave-Allured (Dave Allured)]

Suggestions:

1. In gfortran and other compilers, allocate huge arrays dynamically, not statically, with allocate statements. Yes it makes a difference.

2. Avoid compile time initialization of huge arrays. Along with this, avoid automatic or implicit initialization of large arrays, e.g. automatic zero fill.

3. Investigate the -fstack-arrays option for gfortran. I do not have direct experience, but it sounds like an option provided specifically to assist with large arrays. I think it moves huge arrays away from the heap and onto the stack. It is in the gfortran documentation under 2.9 Options for code generation conventions.

Perhaps try all of these in various combinations.

[fvaccari]

A quick test with -fstack-arrays didn't help, but I'll investigate further and provide an update here if I'll ever succeed... Thanks!

[kencu (Ken)]

Jeremy, copying you in here as this looks like something you might be interested in.

[fvaccari]

Happy to report that after converting to dynamic allocation the largest arrays of the original program, execution proceeds smoothly till the end.

Tested on arm64 Mac (10.12.1, 64GB RAM), Intel Mac (10.6.8 96 GB RAM; 10.14.6 16 GB RAM), CentOS 6 VM hosted on Intel Mac (VirtualBox, 8GB RAM assigned).

Thanks to all who contributed!

[AlanTAMU]

Thank you Dave for pointing out the dynamic storage possibility! Much appreciated!!

Dynamic storage for a large matrix does the trick (at least for me)!

I used the simple (old fashioned) test code below and it worked when run with imax,jmax=50000. Then I used the same approach in my actual code (a finite element code now with dynamic storage for the stiffness matrix) and it worked when static storage failed.

I compiled the code using:

gfortran -o dya -Ofast dya.f

to see if optimization broke anything. It did not.

For the record, I am running MacOS 12.3 on a MacBook Air M1 with 8GBytes. I have both gcc and gcc-devel installed and both seem to work.

program dya
     implicit double precision(a-h,o-z)
     double precision, dimension (:,:), allocatable :: a
     print*, "Enter the size of the array:imax,jmax:"     
     read*, imax,jmax
     allocate (a(imax,jmax))
     call foo(a,imax,jmax)
     deallocate (a)  
     stop
     end
     subroutine foo(a,imax,jmax)
      implicit double precision(a-h,o-z)
     dimension a(imax,jmax)
     do i = 1, imax         
     do j = 1, jmax                
        a(i,j) = dble(i*j)            
     end do      
     end do 
     print*,a(imax,jmax)
     return
     end

[mascguy (Christopher Nielsen)]

Closing, based on the previous two comments. Glad this is resolved for you folks!

[tenomoto (Takeshi Enomoto)]

Dynamic allocation must be the right solution but is not practical for existing code consisted with hundreds of files and thousands of lines (e.g. weather, climate, fluid dynamics, etc.). I was able to compile and run the simple example in #comment:5 with -O (also -O2, -O3, -Ofast, obviously not -O0), but not the weather model at hand. I need to use different optimization levels for some source to compile. The final executable still cause the same dyld error and halts. Other trials I failed include

• -fstack-arrays, -fno-stack-arrays
• All or every options implied by -O listed in ​https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html
• Link with Apple /usr/bin/ld

% gfortran-mp-12 foo.f90
% ./a.out
dyld[86786]: dyld cache '(null)' not loaded: syscall to map cache into shared region failed
dyld[86786]: Library not loaded: /usr/lib/libSystem.B.dylib
Referenced from: <1119BF13-35CB-367A-A471-14B2DF264074> /Users/eno/Development/stack/a.out
Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/libSystem.B.dylib' (no such file), '/usr/lib/libSystem.B.dylib' (no such file, no dyld cache), '/usr/local/lib/libSystem.B.dylib' (no such file)
zsh: abort      ./a.out
% /usr/bin/ld -o a.out foo.o -lSystem -syslibroot $(xcrun -sdk macosx --show-sdk-path) -arch arm64 -L/opt/local/lib/gcc12 -lgfortran 
% ./a.out
dyld[33847]: dyld cache '/System/Library/dyld/dyld_shared_cache_arm64e' not loaded: syscall to map cache into shared region failed
dyld[33847]: Library not loaded: '/usr/lib/libSystem.B.dylib'
  Referenced from: '/Users/eno/Development/stack/a.out'
  Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/usr/local/lib/libSystem.B.dylib' (no such file)
zsh: abort      ./a.out
% gfortran-mp-12 -O foo.f90
% ./a.out
 OK

[Dave-Allured (Dave Allured)]

@tenomoto, dynamic allocation was a workaround to avoid the large memory condition that triggered the original problem. It is not necessarily the complete solution. If dynamic allocation is not practical in your case, then you need to look deeper into the actual error condition. I can not be of much more help because I don't understand how the linker works with Mac SDK's.