Opened 7 years ago

Last modified 4 weeks ago

#38907 assigned enhancement

Resolving libjpeg-turbo + jpeg Port conflicts

Reported by: davidfavor (David Favor) Owned by: larryv (Lawrence Velázquez)
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: vergus@…, ryandesign (Ryan Schmidt), devernay (Frédéric Devernay), floodkoff (Misha Fludkov), jmroot (Joshua Root), MarcusCalhoun-Lopez (Marcus Calhoun-Lopez), mascguy (Chris N)
Port: libjpeg-turbo

Description (last modified by ryandesign (Ryan Schmidt))

Since libjpeg-turbo released there are many Ports that could benefit from it's use, except dependencies in these Ports are for the jpeg Port which conflicts with libjpeg-turbo.

Currently there appears to be no way to specify an "OR dependency"... like...

depends_lib port:jpeg || port:libjpeg-turbo

So is the correct way to fix this to change...

depends_lib port:jpeg

to...

depends_lib path:lib/libjpeg.dylib

which would then pass the dependency check for any Port that installed this library.

Let me know if this is the correct fix.

Thanks.

Change History (29)

comment:2 Changed 7 years ago by davidfavor (David Favor)

Looks like this won't work for libjpeg-turbo, because jpeg9 breaks API/ABI compatibility with jpeg8 and libjpeg-turbo uses the jpeg8 API.

Grumble...

Let me know if this is the correct way to resolve this type of conflict, for possible future work.

Thanks.

comment:3 Changed 7 years ago by ryandesign (Ryan Schmidt)

Cc: openmaintainer removed
Description: modified (diff)
Owner: changed from macports-tickets@… to rmstonecipher@…
Port: libjpeg-turbo added; ffmpeg removed
Type: requestenhancement

If libjpeg-turbo were a drop-in compatible replacement for jpeg (which I thought it was, but which you've pointed out as of jpeg 9 it isn't anymore) then yes the solution would be to change all port:jpeg dependencies in all ports to path:lib/libjpeg.dylib:jpeg.

comment:4 Changed 6 years ago by rmstonecipher@…

Ryan, David,
The libjpeg-turbo commentary on jpeg 9, SmartScale, etc. gives me the impression that the changes to ijg jpeg are non-standard, academic in nature, and not widely-adopted.
Could we not modify the path dependency to depend on libjpeg-turbo by default and let jpeg 9, which seems to be drifting away from standards-compliance, be the alternative library?
Could Portfile voodoo be used to force systems upgrading jpeg-dependent packages to deactivate jpeg 9, install libjpeg-turbo, and rebuild against it?

comment:5 Changed 6 years ago by mf2k (Frank Schima)

Owner: changed from rmstonecipher@… to macports-tickets@…
Version: 2.1.3

comment:6 Changed 5 years ago by vergus@…

Cc: vergus@… added

Cc Me!

comment:7 Changed 5 years ago by tim.bruylants@…

Please do not support libjpeg9! The comment and link that was posted in comment 4 is 100% correct.

Libjpeg9 not only has a changed ABI/API, but it also diverts from the official JPEG standard. Thus, it potentially generates conflicting, and non-standard JPEG files. As far as I know, most of the larger Linux distros also dropped libjpeg9 in favor of libjpeg-turbo.

I, myself, was surprised to find that MacPorts relies on libjpeg9. I am removing this lib from my system as we speak, and suggest everyone from not using it.

Kind regards, Tim.

comment:8 Changed 5 years ago by ryandesign (Ryan Schmidt)

We are now discussing replacing jpeg with libjpeg-turbo on the macports-dev mailing list.

comment:9 Changed 5 years ago by ryandesign (Ryan Schmidt)

Cc: ryandesign@… added

Cc Me!

comment:10 Changed 3 years ago by devernay (Frédéric Devernay)

Cc: devernay added

comment:11 Changed 3 years ago by floodkoff (Misha Fludkov)

Cc: floodkoff added

comment:12 Changed 15 months ago by devernay (Frédéric Devernay)

Is there any update on this?

Ports that depend specifically on the new jpeg9 API (there is very probably none) could keep the dependency on port:jpeg, but most (if not all) ports should depend on path:lib/libjpeg.dylib:jpeg

Major Linux distributions now use libjpeg-turbo instead of libjpeg9 (Fedora, Debian, Mageia, openSUSE).

Official binaries for Mozilla and Chrome also use libjpeg-turbo.

Also note that libjpeg-turbo is now an Official ISO/ITU-T Reference Implementation (see https://jpeg.org/items/20190204_press.html ).

There should be no more discussion on changing depend_libs port:jpeg to depend_libs path:lib/libjpeg.dylib:jpeg.

The only remaining discussion should be whether MacPorts should default to libjpeg-turbo as its default JPEG library. I would vote YES for that, given the very strong support.

comment:13 Changed 15 months ago by mf2k (Frank Schima)

Owner: changed from macports-tickets@… to larryv
Status: newassigned

comment:14 Changed 15 months ago by devernay (Frédéric Devernay)

In fact, one issue with that change is that ports compiled against jpeg9 may link to libjpeg.9.dylib (I just ran into this with the jpeg loading plugin in Qt), but libjpeg-turbo provides libjpeg.8.dylib, so it would also be a good thing to downgrade libjpeg to v8.

Once again, nobody uses the v9 API, which is described by the JPEG itself in https://jpeg.org/items/20190204_press.html as: "a complete implementation of JPEG coming from the Committee itself that also covers coding modes that were only known by a few experts."

comment:15 Changed 11 months ago by dclunie

Any update on this?

I need to build something depends on libjpeg-turbo but also giflib, and the latter depends on jpeg.

This is a showstopper for me and I am going to have to switch to HomeBrew instead of MacPorts if it is not resolved.

comment:16 Changed 11 months ago by kencu (Ken)

  1. it's super-simple to switch a given port as you wish. I have dozens of such ports. Less than two minutes work. Let me know if you are interested.
  2. what did homebrew decide? turbo for everything?

At the very least, seems to me we can change all the jpeg-requiring ports to a path dep, so they can all coexist....

comment:17 Changed 11 months ago by kencu (Ken)

So there are 211 ports that depend on port:jpeg so that's why nobody wants to take this one. Every one will have to be individually tested against turbo, etc.

You say giflib depends on port:jpeg? I'm not seeing that. If you can find the one port that needs the jpeg spec changed, I can probably help with that:

$ port rdeps giflib
The following ports are dependencies of giflib @4.2.3_0+x11:
  xorg-libsm
    pkgconfig
      libiconv
        gperf
    xorg-xtrans
    xorg-libice
      xorg-xorgproto
  xorg-libX11
    xorg-util-macros
    xorg-libXdmcp
    xorg-libXau
    xorg-libxcb
      python37
        xz
          gettext
            ncurses
        bzip2
        expat
        libedit
        libffi
        openssl
          zlib
        sqlite3
        python_select
        python3_select
      xorg-xcb-proto
        autoconf
        automake
        libtool
          xattr
            unzip
        libxml2
          icu
      xorg-libpthread-stubs

comment:18 in reply to:  16 Changed 11 months ago by ryandesign (Ryan Schmidt)

Replying to kencu:

At the very least, seems to me we can change all the jpeg-requiring ports to a path dep, so they can all coexist....

Read comment:3.

comment:19 Changed 11 months ago by kencu (Ken)

Yes -- well I'm certainly not about to test 211 ports against a new jpeg library... I guess dclunie is off to homebrew, then, unless he can come back with the port that is really causing his trouble...

comment:20 Changed 11 months ago by ryandesign (Ryan Schmidt)

It's not about testing, it's just that libjpeg and libjpeg-turbo are not binary compatible so they can't be listed as being so via a path:-style dependency. (It's the same problem with have with openssl: we list openssl and libressl as binary compatible by using a path:-style dep; they're not; we shouldn't do that.)

comment:7 suggests we should switch to libjpeg-turbo as our primary jpeg, making libjpeg9 an option only for ports that need the new features (and naturally moving jpeg9's files so that they don't conflict with libjpeg-turbo's.) We could certainly do that. I'm not going to work on that right now. If somebody else wants to, that's great.

comment:21 Changed 11 months ago by kencu (Ken)

Oh, I see what you're saying .. a path dep should mean you could uninstall one and install the other and all would be hunky-dory without rebuilding anything...not that either could satisfy the need.

Yeah, by that standard, a path dep will never work -- actually would probably quite rarely work for anything, come to think of it.

I took the more liberal view of it, meaning one or the other could be installed.

Perhaps that more liberal view of a path dep is more in keeping with what people actually want.... for example, webkit2-gtk and webkit2-gtk-devel share a path dep, but are not definitely binary compatible --- but it is likely that software could build against one or the other (actually, even that is not particularly guaranteed, but ... ).

comment:22 in reply to:  21 Changed 11 months ago by ryandesign (Ryan Schmidt)

Replying to kencu:

Oh, I see what you're saying .. a path dep should mean you could uninstall one and install the other and all would be hunky-dory without rebuilding anything...not that either could satisfy the need.

Yeah, by that standard, a path dep will never work -- actually would probably quite rarely work for anything, come to think of it.

It works in the original use case: a library, like libpixman or cairo or pango or glib2 or graphviz that maybe adds some functions in a new minor library version but does not remove functions and the major library version does not change. A user can install the -devel version of the port to get new features, and the programs that use the library still work without a rebuild.

I took the more liberal view of it, meaning one or the other could be installed.

Perhaps that more liberal view of a path dep is more in keeping with what people actually want.... for example, webkit2-gtk and webkit2-gtk-devel share a path dep, but are not definitely binary compatible --- but it is likely that software could build against one or the other (actually, even that is not particularly guaranteed, but ... ).

In that scenario, the binaries that the user gets of any programs linking with the library would be broken and would need to be rebuilt from source on the user's system. That's not the user experience we want.

comment:23 in reply to:  20 Changed 11 months ago by devernay (Frédéric Devernay)

Replying to ryandesign:

It's not about testing, it's just that libjpeg and libjpeg-turbo are not binary compatible

jpeg v8 API is fully supported by libjpeg-turbo, and I don't know of any open source software that uses the jpeg v9 API, which is purely research and experimental.

Please read the full explanation here: https://libjpeg-turbo.org/About/Jpeg-9

Most Linux distributions already switched to libjpeg-turbo and it was painless since everyone uses the jpegv8 API.

I'm using the path: approach (depends_lib path:lib/libjpeg.dylib:jpeg) to patch all ports that depend on jpeg, and it works very well!

comment:24 Changed 10 months ago by dclunie

Sorry for the delayed reply - I did not end up switching to Homebrew after all, but just worked around the problem with an Ubuntu Docker build environment that had the libjpeg-turbo and giflib dependencies installed.

That said, I would still like to be able to have both libjpeg (from the IJG) and libjpeg-turbo available so that packages that depended on them could co-exist (and I wouldn't have to mess with Docker).

For that matter there is also the ISO group's code from Tom Richter (http://github.com/thorfdbg/libjpeg), which I understand is also called "libjpeg" even though it has nothing to do with the IJG libjpeg, but which includes a lot of other functionality.

comment:25 Changed 9 months ago by RJVB (René Bertin)

I have been using libjpeg-turbo for about 3 years now myself, with a tweaked port:jpeg that can be co-installed but that acts mostly if not completely as a stub just to satisfy port dependencies (so I haven't kept it up to date). I'm now using libjpeg-turbo 2.0.4 (with a small patch so it registers a compatibility_version of 8 instead of 10, that feature mostly bites us in software from Linux, esp. if it switched from autoconf to cmake :-/) port:jpeg wouldn't be the first to be installed in subprefix, requiring a small patch to ports that really need it (or that want to provide an option to use the reference library).

As others above I have never run into any problems because of a missing v9 API, so I really don't understand what's the hold-up here. Even if port:jpeg provides the reference implementation I think that most users don't give a rat's ass about that while even more will probably appreciate any possible speed improvement in this department.

EDIT: the compatversion tweak above can easily be turned into a tweak that will make current binaries accept the turbo libjpeg dylib without needing to rebuild a single port. A rev-upgrade should then point out immediately if any of the installed ports do use the v9 API, in case anyone wants to be 100% certain about this question.

Last edited 9 months ago by RJVB (René Bertin) (previous) (diff)

comment:26 Changed 9 months ago by jmroot (Joshua Root)

Cc: jmroot added

As has already been stated by Ryan above, if someone were to make a PR switching all jpeg dependencies to libjpeg-turbo and rev bumping, there's no reason it wouldn't be accepted. It's just a matter of doing the work.

comment:27 Changed 9 months ago by MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)

Cc: MarcusCalhoun-Lopez added

comment:28 Changed 9 months ago by devernay (Frédéric Devernay)

What would be the prefered way, then?

depends_lib path:lib/libjpeg.dylib:jpeg

or

depends_lib path:lib/libjpeg.dylib:libjpeg-turbo

or

depends_lib libjpeg-turbo

?

The first and second solution allow switching between jpeg and jpeg-turbo, but it won't work if they have different dylib version numbers.

comment:29 Changed 9 months ago by jmroot (Joshua Root)

depends_lib path:lib/libjpeg.dylib:libjpeg-turbo would be the right way. This would allow switching between libjpeg-turbo and mozjpeg. The jpeg port is not binary compatible with these, and should be changed such that it doesn't install the same files as them (different prefix for most files and maybe a libjpeg.9.dylib link in ${prefix}/lib).

comment:30 Changed 4 weeks ago by mascguy (Chris N)

Cc: mascguy added
Note: See TracTickets for help on using tickets.