qrupdate: +atlas variant fails to build: undefined symbols

[moe.nicholas@…]

Hello,

I am trying to install octave on my PowerBook G4. I have run

$ sudo port selfupdate

and

$ sudo port upgrade outdated

The last dependency to install, qrupdate, fails to build and gives the following output:

$ sudo port install octave
--->  Computing dependencies for octave
--->  Dependencies to be installed: qrupdate
--->  Fetching qrupdate
--->  Attempting to fetch qrupdate-1.1.1.tar.gz from http://superb-east.dl.sourceforge.net/qrupdate
--->  Attempting to fetch qrupdate-1.1.1.tar.gz from http://downloads.sourceforge.net/qrupdate
--->  Verifying checksum(s) for qrupdate
--->  Extracting qrupdate
--->  Applying patches to qrupdate
--->  Configuring qrupdate
--->  Building qrupdate
Error: Target org.macports.build returned: shell command failed
Log for qrupdate is at: /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_ports_math_qrupdate/main.log
Error: The following dependencies failed to build: qrupdate
Error: Status 1 encountered during processing.
To report a bug, see <http://guide.macports.org/#project.tickets>
$

I am attaching the log file cited above.

[jmroot (Joshua Root)]

The symbols that are undefined here are present in /opt/local/lib/liblapack.dylib on my system (10.6/x86_64).

[penyuan@…]

The same thing happens on my MacBook Pro running Mac OS X 10.6.4 when trying to install Octave. Is there a workaround?

[penyuan@…]

Cc Me!

[stefan@…]

Same on my iBook G4 with 10.5.8. liblapack.dylib is present but not recognized.

[stefan@…]

Cc Me!

[michaelld (Michael Dickens)]

On my 10.6.4 x86_64 box, the undefined symbols are actually in -lptf77blas (either .a or .dylib) -- which is interesting because that library isn't directly referenced during the link stage (nor is the linking told to ignore undefined symbols), but the linking still works. I'm attached a patch that might work; please try it out.

[michaelld (Michael Dickens)]

The Octave issue is on another ticket already. For -devel, that's #26640. IDK which one it is for octave alone, but I'm sure it's there.

[michaelld (Michael Dickens)]

@stefan: Can you attach a debug-log of your install, to see if it's the same issue as for this ticket?

[michaelld (Michael Dickens)]

"svn diff" of 'qrupdate' Portfile, to add in missing atlas symbols

[michaelld (Michael Dickens)]

Cc Me!

[stefan@…]

main.log file of my failed qrupdate build. iBook G4 OS X 10.5.8

[mfuhrer@…]

Ran into this link problem this evening on my Powerbook G4 with Mac OS X 10.5.8. Tried Michael's patch with no success, as I don't have -lptf77blas in /opt/local/lib. Then I found -lf77blas which does define the missing symbols. Modified Michael's patch and was able to link and install qrupdate successfully.

[mfuhrer@…]

Modified patch with -lf77blas

[mfuhrer@…]

Cc Me!

[AlonzoQuixote@…]

Cc Me!

[jsnodgra@…]

Cc Me!

[ejtttje@…]

I'm seeing this now following an update to atlas (#34295) which builds the blas libraries in question. I also had to add -latlas.

The -ptf77blas and -ltatlas give multi-threaded usage, whereas -lf77blas and -latlas give single-threaded libraries. I'm not sure which is more appropriate here.

I still wound up running into issues with __gfortran_transfer_character_write and __gfortran_transfer_integer_write being undefined, which is supposedly fixed by setting the linker/loader to use gfortran, but I couldn't figure that out. I wound up adding -undefined dynamic_lookup to get around this, which probably isn't the right solution.

[ejtttje@…]

Cc Me!

[Veence (Vincent)]

The dylibs automatically built by atlas 3.9.75 differ from those that were (manually) built by 3.9.47. They are named libtatlas.dylib and libsatlas.dylib. You must look after the undefined symbols using nm then maybe patch qrupdate accordingly.

[nabble@…]

I had a similar problem trying to install octave, link failing on qrupdate, but I had to tweak my Portfile differently to get it to build - attached as qrupdate_Portfile3.diff

[ryandesign (Ryan Carsten Schmidt)]

Vincent, is the patch attached by makuch.org correct—add "-latlas -lsatlas -ltatlas -lf77blas -lptf77blas"?

I've rearranged things a bit in my update to 1.1.2 so you may want to keep that in mind when preparing a fix for this issue as well.

[nabble@…]

I seriously doubt it's correct, but it worked for me :-) so I just thought I'd share it with you guys knowing you would fix it correctly.

[Veence (Vincent)]

-latlas -lf77blas -lptf77blas is redundant with -libsatlas and -libtatlas. The first three are static archives, and the last two dynamic libraries. Then you can choose between satlas or tatlas whether you need threaded Atlas or not. So, to wrap up, only one of libsatlas or libtatlas is needed.

[mauricio_ville@…]

Cc Me!

[MacPorts@…]

Cc Me!

[m1cha3ld0m1n0@…]

Cc Me!

[enderw88@…]

I was able to get qrupdate to build and install by

1. cd'ing into the working directory /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/math/qrupdate/work/qrupdate-1.1.1
2. executing sudo make solib.
3. executing sudo port install qrupdate.

[m1cha3ld0m1n0@…]

Using enderw88's steps, was able to build on 10.5.8. Thanks!

[ryandesign (Ryan Carsten Schmidt)]

enderw88, that's not a fix. When you do that, you're bypassing the PREFIX, BLAS and LAPACK arguments MacPorts would have set for you, so at that point you're not actually using atlas, plus the library will get the wrong install_name. So you may as well install qrupdate without atlas support at that point.

sudo port selfupdate
sudo port clean qrupdate
sudo port install qrupdate -atlas

[enderw88@…]

Replying to ryandesign@…:

enderw88, that's not a fix. When you do that, you're bypassing the PREFIX, BLAS and LAPACK arguments MacPorts would have set for you, so at that point you're not actually using atlas, plus the library will get the wrong install_name. So you may as well install qrupdate without atlas support at that point.

sudo port selfupdate
sudo port clean qrupdate
sudo port install qrupdate -atlas

Thanks, at this point I just really want to get Octave built. But I can see why this isn't a real solution.

[sewebster@…]

Cc Me!

[jerome.lelong@…]

Add blas dependency for +atlas variant

[jerome.lelong@…]

qrupdate does not build with variant +atlas because of missing symbols at link actually provided by Blas. It seems that the patch qrupdate_Portfile_4.diff solves the issue at least if atlas was installed with gcc45. If atlas was installed with variant +gcc46, then you should consider adding the corresponding variant to qrupdate to achieve the build.

[ben.pts@…]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #34654.

[weasel@…]

It appears a diff four fixes the problem I noted in #34654 with qrupdate, but forces atlas +gcc46 be installed, which oddly enough completes without gcc46 being installed. There is still no /opt/local/lib/libsatlas.dylib, maybe that's intentional. I haven't yet tested if octave ultimately works correctly.

[mojca (Mojca Miklavec)]

Cc Me!

[jovo@…]

Cc Me!

[beunjb@…]

Octave works with the diff patch

cd $(port dir qrupdate)
sudo patch -p0 < ~/Downloads/qrupdate_Portfile_4.diff

Thanks for the good work.

[rjfergus@…]

Replying to beunjb@…:

Octave works with the diff patch

cd $(port dir qrupdate)
sudo patch -p0 < ~/Downloads/qrupdate_Portfile_4.diff

Thanks for the good work.

Hi,

I have the patch file in my $(port dir qrupdate) directory but when I run the patch command above, I get "can't find file to patch at input line 3" and "Perhaps you used the wrong -p or --strip option?". I notice in the diff file that there are some strange tabs but I don't know the syntax ...

[lawrence.ong@…]

Cc Me!

[lawrence.ong@…]

Using the forth patch, qrupdate and octave both install without error. But when launching octave, I get the following error message:

dyld: Library not loaded: /opt/local/lib/libptcblas.dylib
  Referenced from: /opt/local/lib/libarpack.2.dylib
  Reason: image not found
Trace/BPT trap: 5

[Veence (Vincent)]

The patch should read: BLAS="-L${prefix}/lib -ltatlas" and nothing else behind.

Arpack needs rebuilding too. The Atlas dynamic libraries have changed into libsatlas (single threaded) and libtalas (threaded).

[Veence (Vincent)]

I have committed r93765 that should fix this bug. GCC46 is now default, like atlas, to avoid messing up gfortran libraries. Please test and report.

[ryandesign (Ryan Carsten Schmidt)]

We want all ports having gcc variants to default to the same version. gcc45 is the version we have chosen at this time. If you want to change that default for all ports, you should start a discussion about it on the mailing list.

[Veence (Vincent)]

I wasn’t aware of this. Then you can make the default gcc depend on whether the +atlas variant is set or not. gcc46 in the former case, gcc45 in the latter.

[ryandesign (Ryan Carsten Schmidt)]

Why shouldn't atlas default to gcc45 as well?

[Veence (Vincent)]

Because atlas is all about performance and a gcc46 compiled atlas performs 10 to 15 % better than a gcc45 one. On the other hand, gcc47 improvements over gcc46 (at this time) are moot.

[anddam (Andrea D'Amore)]

Replying to vince@…:

gcc46 compiled atlas performs 10 to 15 % better than a gcc45 one.

That's not a really big enhancement, I vote for keeping the default at gcc45 like other fortran-needing ports.

If an user really needs the speed bonus he/she can enable +gcc46, speaking in a broad sense I'd keep dependencies more streamlined.

[anddam (Andrea D'Amore)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

The reason we default everything to the same gcc version is that subtle hard-to-track-down problems can occur if programs linked with different versions of the C library (i.e. from different versions of the gcc ports) are linked together. I don't know if that affects atlas or not.

[jerome.lelong@…]

If atlas was compiled with gcc46, then qrupdate must alo be compiled with gcc46 otherwise linking fails with missing Fortran symbols. Hence, a gcc46 variant should be added to qrupdate Portfile even if gcc45 remains the default.

[Veence (Vincent)]

The update I have committed have removed obsolete gcc43/44 variants in favor of gcc46/47.

[Veence (Vincent)]

Replying to and.damore@…:

Replying to vince@…:

gcc46 compiled atlas performs 10 to 15 % better than a gcc45 one.

That's not a really big enhancement

That’s your opinion. In the field of HPC, 10% is a lot, especially when all you have to do is change compiler. It’s about half an hour win every four hours. If a user is not concerned with Atlas, he can just opt out by selecting the no_atlas variants and get standard Accelerate framework performance. If he cares about Atlas, then it’s probably to squeeze the most out of his hardware.

[mf2k (Frank Schima)]

@vince: Any user concerned about tweaking their machine's performance can - and should - be using variants.conf to set their gcc version to 4.6 or 4.7 anyway. The vast majority of Macports users don't know or care about this. Instead they will (rightly) complain that they have two versions of gcc installed. So gcc45 should be used as the default to keep in line with the other fortran ports.

[Veence (Vincent)]

Well, I only half agree because the people who are likely to use Atlas, Octave, and so forth are probably only those in need of processing power, and not the typical Macports user, if that notion makes any sense. But okay, I gracefully abide by the majority’s opinion. I’ll update Atlas accordingly. Anyhow, there is always this major flaw in Atlas whereby AVX instructions cannot be used, and that alone halves the performance of some kernels…

[anddam (Andrea D'Amore)]

Replying to vince@…:

That’s your opinion. In the field of HPC, 10% is a lot, especially when all you have to do is change compiler.

Yes, it is my opinion that a 0.1 factor is an order of magnitude smaller than the total time and it's not meaningful to common users. That we should focus on people doing intensive computing is yours.

I believe that anyone doing HPC has the responsibility to tune his/her system.

If a user is not concerned with Atlas, he can just opt out by selecting the no_atlas variants and get standard Accelerate framework performance. If he cares about Atlas, then it’s probably to squeeze the most out of his hardware.

He could be using atlas just for a dependency.

Back to this ticket

This ticket is messy, it's not clear if users after comment:12 are experiencing the same issue as reporter.

I stumbled into a similar one with different symbols' name after the name change in single- and multi-threaded atlas library.

I'm attaching a new patch that adds -latlas to BLAS and bumps revision to force rebuild.

Please anyone who had the issue confirm it's working so we can close this ticket.

[lawrence.ong@…]

Replying to vince@…:

The patch should read: BLAS="-L${prefix}/lib -ltatlas" and nothing else behind.

Arpack needs rebuilding too. The Atlas dynamic libraries have changed into libsatlas (single threaded) and libtalas (threaded).

Thanks. Rebuilding arpack solves the following problem:

dyld: Library not loaded: /opt/local/lib/libptcblas.dylib
  Referenced from: /opt/local/lib/libarpack.2.dylib
  Reason: image not found
Trace/BPT trap: 5

[anddam (Andrea D'Amore)]

Vince had already committed r93765 by the time I last replied, I was mistakenly using a previous version of Portfile.

I've switched default variant back to +gcc45 and bumped revision in r93783, r93785.

Closing the ticket as fixed.

[jerome.lelong@…]

There is still a conflict in the Portfile between gcc45 and gcc46. The qrupdate_Portfile_5.diff patch solves the issue.

[jerome.lelong@…]

Fix conflict between gcc45 and gcc46

[anddam (Andrea D'Amore)]

Replying to jerome.lelong@…:

The qrupdate_Portfile_5.diff patch solves the issue.

The patch was not made from current Portfile, anyway I committed the change in r93792.

Ticket closed.

[anddam (Andrea D'Amore)]

Replying to and.damore@…:

I've switched default variant back to +gcc45 […] in r93783

That's actually r93782.

[sam.petulla@…]

Hi

I'm having all kinds of problems installing qrupdate and cannot install the patch. I get the following error after running: sudo patch -p0 < ~/Downloads/qrupdate_Portfile_4.diff

patching file /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/math/qrupdate/Portfile
Hunk #1 FAILED at 69.
1 out of 1 hunk FAILED -- saving rejects to file /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/math/qrupdate/Portfile.rej

[ryandesign (Ryan Carsten Schmidt)]

This ticket is closed and fixed and the patch was already committed 11 months ago; you do not need to apply it manually. Just sudo port selfupdate to receive the changes, as usual. If you are unable to install the port, file a new ticket.