Installing recent versions of yq or wget installs libnetpbm which has a full X11/UI list of dependencies

[tux-o-matic (Benjamin Merot)]

For some reason, trying to upgrade wget and yq which are both CLI tools, I see a full log of dependencies related to GUI libraries. Such as: (if I list them with "port reclaim"):

 ghostscript  @10.03.0_0+x11
 xorg-libXext  @1.3.6_0
 xorg-libXt  @1.3.0_0
 xorg-libX11  @1.8.8_0
 xorg-libxcb  @1.16.1_0+python312
 xorg-xcb-proto  @1.16.0_0+python311
 xorg-util-macros  @1.20.0_0
 xorg-xtrans  @1.5.0_0
 xorg-libXau  @1.0.11_0
 xorg-libsm  @1.2.4_0
 xorg-libice  @1.1.1_0
 xorg-xorgproto  @2023.2_0
 xorg-libXdmcp  @1.1.5_0

[jmroot (Joshua Root)]

For wget the dependency comes in via gpgme: gnupg2 -> openldap -> (build-time dependency only) groff -> netpbm

OTOH, yq has a build time dependency on pandoc, which depends on pretty much everything.

[herbygillot (Herby Gillot)]

That was added in this PR here: ​https://github.com/macports/macports-ports/pull/22893

A lot of projects use pandoc to generate documentation, but it's definitely a very heavy dependency.

[ryandesign (Ryan Carsten Schmidt)]

It is reasonable to have pandoc as a dependency of a -docs port that only installs documentation (assuming pre-generated documentation is not available) but I don't think it's reasonable to require it for a regular port.

yq should install a pre-generated manpage. The yq project publishes that at ​https://github.com/mikefarah/yq/releases/download/v4.43.1/yq_man_page_only.tar.gz

[ryandesign (Ryan Carsten Schmidt)]

Replying to jmroot:

For wget the dependency comes in via gpgme: gnupg2 -> openldap -> (build-time dependency only) groff -> netpbm

groff's netpbm dependency has been there for 13 years however ​the developer of groff says "With the distributed tarballs, there is *no* dependency on Netpbm!"

[tux-o-matic (Benjamin Merot)]

Thanks for finding the root cause of this. The experience is even more painful on one of my x86 systems which fails on the compilation of netpbm. So not only all those build time dependencies for popular CLI tools feel unnecessary, but they introduce a higher risk of breaking the upgrade of other Ports.

[ryandesign (Ryan Carsten Schmidt)]

If you're not sure of the cause of that netpbm build failure please file a separate ticket about it so I can take a look.

[SaintBol]

Same issue with wget2 found today (wget2 @2.1.0_1), wanting groff, wanting netpbm, and so on.

[lcvisser (Ludo Visser)]

Attached is is the full dependency list that I was presented with today, after installing Sequoia and trying to reinstall ports (from source). More often than not one of these packages fails at some point in time (today it was py312-pluggy). It seems that at least 90% of these dependencies should not be there.

I use wget a lot and with every update I spend probably a day to get it to work again. I'd be very happy to spend that time in contributing to a solution (cutting down the dependency list) instead, but not sure where to start. Some pointers?

[lcvisser (Ludo Visser)]

List of dependencies for wget

[ryandesign (Ryan Carsten Schmidt)]

Replying to lcvisser:

More often than not one of these packages fails at some point in time (today it was py312-pluggy).

I can't find any current MacPorts bug reports about py312-pluggy. Please file individual tickets for each port that fails to build so that it can be investigated and fixed.

It seems that at least 90% of these dependencies should not be there.

That's not an actionable suggestion. If you have specific knowledge of a dependency that is not needed somewhere, please file a bug report or pull request with appropriate evidence. It's certainly possible that a dependency used to be needed by some port but then it was updated to a new version that doesn't need it anymore and we didn't notice. Or a dependency could have been added erroneously. But for each dependency that you believe a port doesn't need, some investigation would have to happen by you or someone else to confirm it and take the appropriate action. For example, if a dependency is optional and we want to remove it, we additionally must ensure that it does not get used if it happens to be installed, otherwise builds are not reproducible.

I use wget a lot and with every update I spend probably a day to get it to work again.

I'm sorry to hear that; that's obviously not the experience we want users to have. We can't fix problems we don't know about, so report each problem you find, presuming reports do not already exist for those problems.

I'd be very happy to spend that time in contributing to a solution (cutting down the dependency list) instead, but not sure where to start. Some pointers?

Well Josh said:

Replying to jmroot:

For wget the dependency comes in via gpgme: gnupg2 -> openldap -> (build-time dependency only) groff -> netpbm

And I replied:

Replying to ryandesign:

groff's netpbm dependency has been there for 13 years however ​the developer of groff says "With the distributed tarballs, there is *no* dependency on Netpbm!"

I can remove the netpbm dependency, if it's truly not needed, however that still leaves ghostscript which also depend on X11 when its x11 variant is selected, which it is by default.

And I'm not convinced that netpbm is not needed. The groff Portfile says the reason the dependencies are there is because they are specified in groff's README, and indeed they are:

Ghostscript is required for creation of PDF and (X)HTML output.
Production of (X)HTML furthermore demands tools from the 'netpbm' and
'psutils' packages.

Perhaps a new discussion needs to be had with the developer of groff to get to the bottom of it. Is the README out of date? Or, if the README remains accurate, is it the case that we have no need for MacPorts groff to produce (X)HTML and we can therefore instruct it not to use netpbm or psutils? If you'd like to start that discussion with the developer, or if you find an existing discussion that answers the question, please provide a link to it here.

[lcvisser (Ludo Visser)]

Replying to ryandesign:

I can't find any current MacPorts bug reports about py312-pluggy. Please file individual tickets for each port that fails to build so that it can be investigated and fixed.

The problem is not in py312-pluggy itself but elsewhere; py312-pluggy just happened to be the first one to trip over it. I'm still trying to figure out what is going on exactly.

That's not an actionable suggestion. If you have specific knowledge of a dependency that is not needed somewhere, please file a bug report or pull request with appropriate evidence. It's certainly possible that a dependency used to be needed by some port but then it was updated to a new version that doesn't need it anymore and we didn't notice. Or a dependency could have been added erroneously. But for each dependency that you believe a port doesn't need, some investigation would have to happen by you or someone else to confirm it and take the appropriate action. For example, if a dependency is optional and we want to remove it, we additionally must ensure that it does not get used if it happens to be installed, otherwise builds are not reproducible.

Aside from the X11 dependencies I also see libusb, 6 versions of docbook-xml, a ton of Python and Perl packages (while wget is written in C). I suspect that a lot of these dependencies get pulled in for the build process, perhaps for documentation generation. Perhaps "should not be there" is not the best wording from my side, but it would be nice if there was a way to "opt-out" of them, e.g. via a variant.

I'm sorry to hear that; that's obviously not the experience we want users to have. We can't fix problems we don't know about, so report each problem you find, presuming reports do not already exist for those problems.

I'm in a non-standard situation (corporate machine, no administrator rights, illegal group names with spaces, etc.) and build everything from source, so a lot of the problems I'm seeing are not MacPorts fault. I'm therefore hesitant to file a bug reports, unless I'm absolutely sure it's not due to my edge case configuration.

I'd be very happy to spend that time in contributing to a solution (cutting down the dependency list) instead, but not sure where to start. Some pointers?

Well Josh said:

Replying to jmroot:

For wget the dependency comes in via gpgme: gnupg2 -> openldap -> (build-time dependency only) groff -> netpbm

And I replied:

Replying to ryandesign:

groff's netpbm dependency has been there for 13 years however ​the developer of groff says "With the distributed tarballs, there is *no* dependency on Netpbm!"

I can remove the netpbm dependency, if it's truly not needed, however that still leaves ghostscript which also depend on X11 when its x11 variant is selected, which it is by default.

And I'm not convinced that netpbm is not needed. The groff Portfile says the reason the dependencies are there is because they are specified in groff's README, and indeed they are:

Ghostscript is required for creation of PDF and (X)HTML output.
Production of (X)HTML furthermore demands tools from the 'netpbm' and
'psutils' packages.

Perhaps a new discussion needs to be had with the developer of groff to get to the bottom of it. Is the README out of date? Or, if the README remains accurate, is it the case that we have no need for MacPorts groff to produce (X)HTML and we can therefore instruct it not to use netpbm or psutils? If you'd like to start that discussion with the developer, or if you find an existing discussion that answers the question, please provide a link to it here.

Is it possible to use the port utility to systematically trace dependencies? I'd be interested to learn what the dependency tree looks like and if there are port variants that can cut down the tree a bit (seem like installing Ghostscript without the x11 variant would be a good start...)

I'm not trying to be a smart ass. I've been using macports since my first Mac (Snow Leopard) and 99/100 times it works flawlessly. I'm aware that I'm in an unusual configuration which makes my life a bit harder than most users, but I'm hoping to see if I can contribute back for the few others (e.g. #69650) that are facing similar problems.

[Dave-Allured (Dave Allured)]

Replying to lcvisser:

Is it possible to use the port utility to systematically trace dependencies?

port rdeps wget for example.

[Dave-Allured (Dave Allured)]

openldap -> (build-time dependency only) groff

The openldap portfile says "The openldap build uses soelim from groff, and newer OS versions do not provide groff as part of the base OS install." This brings in an entire typesetting system, including netpbm, to build openldap. Is this really necessary? What is "soelim"? Can there be an easier shortcut to satisfy the "soelim" need?

[ryandesign (Ryan Carsten Schmidt)]

Replying to Dave-Allured:

The openldap portfile says "The openldap build uses soelim from groff, and newer OS versions do not provide groff as part of the base OS install." This brings in an entire typesetting system, including netpbm, to build openldap. Is this really necessary? What is "soelim"? Can there be an easier shortcut to satisfy the "soelim" need?

According to its manpage, "GNU soelim is a preprocessor for the groff(7) document formatting system." Whether netpbm is necessary for groff is one of the topics to be investigated in this ticket, for example by communicating with the developer of groff.

According to the build/man.mk file in the openldap distribution, it uses soelim to create its manpages. If you want to see what happens without soelim available, and see if you can make the port build another way, you can remove the groff dependency from the port and either use trace mode (if it works on your OS version and architecture) or deactivate groff; I can't test this because I use macOS 12 which still includes soelim. If you're thinking of making it work by disabling the manpages, that's probably not what we want; people like to be able to look in manpages to learn how to use software.

According to a search on the ports web site, heirloom-doctools is another port that provides an implementation of soelim. I don't know if it is sufficiently compatible with the GNU groff implementation to be usable for the ports that currently depend on the groff port.

[Dave-Allured (Dave Allured)]

Replying to ryandesign:

Replying to Dave-Allured:

What is "soelim"?

According to its manpage, "GNU soelim is a preprocessor for the groff(7) document formatting system."

Thanks for bringing me up to speed on that.

Whether netpbm is necessary for groff is one of the topics to be investigated in this ticket, for example by communicating with the developer of groff.

According to the build/man.mk file in the openldap distribution, it uses soelim to create its manpages.

It looks like openldap and wget use the same troff/texinfo source format for man pages. wget builds them with the lighter weight texinfo port. Therefore I suggest seeing whether texinfo can be substituted for the groff dependency in openldap. groff has 93 recursive dependencies in MacPorts. texinfo has only 15. There would need to be some command line changes in the build files.

Upstream openldap may have already considered this. It would be worth asking.

[ryandesign (Ryan Carsten Schmidt)]

Replying to lcvisser:

Aside from the X11 dependencies I also see libusb,

The first path to get to the libusb dependency from wget is that wget depends on gpgme which depends on gnupg2 which depends on libusb-compat which depends on libusb. I have not attempted to investigate whether all of those dependencies are actually used for anything useful. There may be more paths to a libusb dependency; port rdeps only shows the first one, unless the --full flag is used, which I never do because in my opinion it produces an unusably large volume of output.

6 versions of docbook-xml,

wget depends on libproxy which depends on vala which depends on docbook-xml which depends on the several versions of docbook-xml. Possibly, vala could be changed to depend on the specific version of docbook-xml that it requires, but I don't think that will result in a large disk space or time savings.

a ton of Python and Perl packages (while wget is written in C).

Many of the perl dependencies come from git. wget -> libproxy -> vala -> graphviz -> gd2 -> libheif -> aom -> git.

aom's git dependency is questioned in #66061.

git used to be written in perl and thus required various perl modules but over time most of its functionality has been rewritten in C. You can opt out of the remaining perl functionality of the git port by installing it while deselecting the perl5_34 variant. I've started doing that on the MacPorts buildbot worker machines as I too was annoyed by having to install a bunch of perl modules, and then wrangle them when the default version of perl in MacPorts changed. It's worth investigating, in a separate ticket, if git's remaining perl functionality could be moved to a separate port or subport.

Various ports in the dependency chain have dependencies on python modules. For example, meson is a build system written in python. wget -> libproxy -> vala -> glib2 -> meson.

vala is a programming language. libproxy depends on vala so that it can provide language bindings that can be used to access libproxy from vala programs. It's worth investigating, in another ticket, if those language bindings could be in a separate port or subport.

it would be nice if there was a way to "opt-out" of them, e.g. via a variant.

You have to be careful with moving functionality to variants because ports can't depend on variants of other ports (see #126) (other than by using the active_variants 1.1 portgroup, which requires manual user intervention and thus cannot work on the buildbot machines). Ports must therefore include all functionality that any other port might depend on. Where possible, optional functionality that only some users or other ports might need should be moved to subports. However, some build systems make this difficult or impossible.

I'm in a non-standard situation (corporate machine, no administrator rights, illegal group names with spaces, etc.) and build everything from source, so a lot of the problems I'm seeing are not MacPorts fault. I'm therefore hesitant to file a bug reports, unless I'm absolutely sure it's not due to my edge case configuration.

Non-root MacPorts installations are supposed to be supported, though because they're not the default problems do creep in from time to time, which we do want to know about. Illegal group names with spaces are annoying but we have made changes to MacPorts in the past to accommodate them so I would not be opposed to seeing bug reports about such problems so they can be fixed. And often, by building from source you might simply be the first to run into a problem that others haven't seen yet because they received a binary we built before some problem existed, so those reports are valuable as well.

I'm not trying to be a smart ass. I've been using macports since my first Mac (Snow Leopard) and 99/100 times it works flawlessly. I'm aware that I'm in an unusual configuration which makes my life a bit harder than most users, but I'm hoping to see if I can contribute back for the few others (e.g. #69650) that are facing similar problems.

No worries; we are happy to have bug reports and especially fixes.

#69650, as far as I could tell, is specific to that user's system; it is not a general problem with non-root installations, for example, since I couldn't reproduce the problem.

[ryandesign (Ryan Carsten Schmidt)]

Replying to Dave-Allured:

It looks like openldap and wget use the same troff/texinfo source format for man pages. wget builds them with the lighter weight texinfo port. Therefore I suggest seeing whether texinfo can be substituted for the groff dependency in openldap.

I don't see any way for that to be possible. troff and texinfo are not the same format. Each format needs to be manipulated by tools that understand that format.

wget distributes its documentation in texinfo (texi) format. Its build system uses makeinfo to convert the texinfo file to info format for use with the info command. makeinfo used to be included with macOS; on older macOS versions it will use the macOS copy of makeinfo if the texinfo port is not installed. wget's build also uses a bundled perl script called texi2pod.pl to convert the texinfo file to perl's pod documentation format, and then uses perl's pod2man program to convert it to *roff format, which is what manpages are and what the man program reads.

openldap, on the other hand, doesn't install documentation in info format at all, only in *roff format. It distributes its documentation in *roff format in fact, but the *roff files it distributes are not the final files users will see in the man program; rather, they're templates that still need some processing. There are placeholders to be replaced with real values (like the openldap version number) which the build system does using sed, and include statements which need to be replaced with the contents of the files being included, which the build system does using soelim.

[Dave-Allured (Dave Allured)]

Replying to ryandesign:

Replying to Dave-Allured:

It looks like openldap and wget use the same troff/texinfo source format for man pages. wget builds them with the lighter weight texinfo port. Therefore I suggest seeing whether texinfo can be substituted for the groff dependency in openldap.

I don't see any way for that to be possible. troff and texinfo are not the same format. Each format needs to be manipulated by tools that understand that format.

My hasty mistake, sorry. Now I see that your previous suggestion of ​heirloom-doctools is a good candidate. I also find ​mandoc. Both are current ports in Macports. Both are extremely lightweight, and both provide soelim for possible replacement of groff as a wget dependency. I would try mandoc first since it seems to be more recently maintained, and builds are working better.

[Dave-Allured (Dave Allured)]

Replying to Dave-Allured:

for possible replacement of groff as a wget dependency

Correction. For possible replacement of groff as an openldap dependency.

[Dave-Allured (Dave Allured)]

It looks like the lightweight mandoc will replace the groff dependency in openldap. The number of openldap recursive dependencies drops from 103 to 33. ​PR #26700 submitted.

[Dave-Allured (Dave Allured)]

In 5a5a25e19863fbbceaa81c9cf9252c7f5cd99b07/macports-ports (master):

openldap: Reduce dependencies (​https://github.com/macports/macports-ports/pull/26700)

• Replace heavy port groff with lightweight mandoc
• Needed only to build man pages with "soelim"
• Recursive dependencies for openldap reduced from 103 to 33

See: #69601

[Dave-Allured (Dave Allured)]

groff is used by other ports. Changing openldap did not make groff go away, so the load for wget did not get any lighter!

This openldap experience suggests that man page builders may be an easy target for more dependency reduction. I think I will next look at xorg-libXaw, and see why it wants groff. I also wonder whether libproxy -> vala could be lightened.

@lcvisser would you like to pick up on any of these?

[lcvisser (Ludo Visser)]

Replying to Dave-Allured:

groff is used by other ports. Changing openldap did not make groff go away, so the load for wget did not get any lighter!

This openldap experience suggests that man page builders may be an easy target for more dependency reduction. I think I will next look at xorg-libXaw, and see why it wants groff. I also wonder whether libproxy -> vala could be lightened.

@lcvisser would you like to pick up on any of these?

I'd be interested in looking into the vala dependency, but don't expect such lightning-fast progress as you made the last 2 days! I'm not so familiar (yet) with the macports internals.

[ryandesign (Ryan Carsten Schmidt)]

Replying to Dave-Allured:

This openldap experience suggests that man page builders may be an easy target for more dependency reduction. I think I will next look at xorg-libXaw, and see why it wants groff.

The groff dependency was added to the xorg-libXaw port back in version 1.0.7 because the version of groff on Tiger didn't work right.

As far as I can tell, as of version 1.0.8, ​libxaw doesn't use groff anymore so the dependency should be removed. It instead uses xmlto and other tools to build documentation, but only when the port's +docs variant is selected, which it isn't by default.

I also wonder whether libproxy -> vala could be lightened.

As I said, it might be possible to move libproxy's vala bindings to a separate port or subport.

[ryandesign (Ryan Carsten Schmidt)]

In af2f673c4cae152da1771ab779a2b29180649492/macports-ports (master):

xorg-libXaw: Remove groff dependency

groff is not used anymore as of libXaw 1.0.8.

See: #69601

Also use path dep for pkgconfig, disable silent rules, and suppress
strchr implicit function declaration warning.

[Dave-Allured (Dave Allured)]

Thanks, Ryan! Finally the number of recursive deps for wget drops from 428 to 417. Some of the dropped ones are groff, ghostscript, and the sometimes problematical netpbm. More, anyone?

Best of luck with libproxy and vala!

[ryandesign (Ryan Carsten Schmidt)]

Replying to Dave-Allured:

and the sometimes problematical netpbm.

In what way is netpbm a problem, other than bringing in more dependencies?

[tux-o-matic (Benjamin Merot)]

I'd just like to thanks people on this old ticket of mine who took the time to dig into the problem to fix it.

[jleroy (Jonathan Leroy)]

Replying to ryandesign:

wget -> libproxy -> vala -> glib2 -> meson.

vala is a programming language. libproxy depends on vala so that it can provide language bindings that can be used to access libproxy from vala programs. It's worth investigating, in another ticket, if those language bindings could be in a separate port or subport.

Opened a ticket regarding this issue.

[lcvisser (Ludo Visser)]

Replying to Dave-Allured:

Thanks, Ryan! Finally the number of recursive deps for wget drops from 428 to 417. Some of the dropped ones are groff, ghostscript, and the sometimes problematical netpbm. More, anyone?

Best of luck with libproxy and vala!

With the vala bindings removed from libproxy, I now have 246 reported dependencies for wget.

[jleroy (Jonathan Leroy)]

Replying to lcvisser:

With the vala bindings removed from libproxy, I now have 246 reported dependencies for wget.

Thank you for your work!

The biggest dependency now seems to be gpgme, with 131 ports needed. gpgme and libmetalink dependency are required to enable Metalink support in wget.

However:

• ​Upstream Metalink support is enabled only if libmetalink is available on the target system,
• Metalink support in wget seems to be disabled on most Linux distributions (Debian, Ubuntu and RHEL at least),
• Metalink support in cURL have been removed upstream in 2021 due to multiples ​security issues.

So maybe the best solution is to just drop Metalink support, or at least add a metalink variant, disabled by default ?

[lcvisser (Ludo Visser)]

I've created a new ticket for making metalink a variant for wget.

[lcvisser (Ludo Visser)]

In 24bb7c412c46903d045938568ad38e91d6428ec6/macports-ports (master):

wget: Move metalink support to a variant

In order to reduce the dependencies of wget, make metalink support
optional via a variant. Currently, wget is the only port that depends on
libmetalink and metalink support is disable on most Linux distributions
in wget and curl.

See: #69601

Closes: #71523

[lcvisser (Ludo Visser)]

Replying to ryandesign:

yq should install a pre-generated manpage. The yq project publishes that at ​https://github.com/mikefarah/yq/releases/download/v4.43.1/yq_man_page_only.tar.gz

I've created ​https://github.com/macports/macports-ports/pull/27065 This brings the number of dependencies for yq down to 1: go

The previous PRs brought the wget dependencies down to 261, of which 120 are various p5.34-*.

In my opinion, this ticket can be closed if the above PR is accepted, unless someone sees more room for improvement.

[Ludo Visser <ludo.visser@…>]

In 7aaa4336391497688bdf7f6321c8bcf60ddc8aac/macports-ports (master):

yq: use pre-generated man page

To avoid the dependencies of yq, use the pre-generated manpage from the
upstream project to avoid the dependency on pandoc.

See #69601

Closes: #69601
Closes: #71536