qgis3 broken by leveldb update

[gwright83]

qgis3 has been broken by the recent leveldb update (#56288).

The error:

:info:build /opt/local/libexec/qt5/bin/moc -DQT_CONCURRENT_LIB -DQT_CORE_LIB -DQT_DEPRECATED_WARNINGS -DQT_DISABLE_DEPRECATED_BEFORE=0x050800 -DQT_GUI_LIB -DQT_MULTIMEDIAWIDGETS_LIB -DQT_MULTIMEDIA_LIB -DQT_NETWORK_LIB -DQT_NO_CAST_TO_ASCII -DQT_NO_DEBUG -DQT_NO_FOREACH -DQT_POSITIONING_LIB -DQT_PRINTSUPPORT_LIB -DQT_QMLMODELS_LIB -DQT_QML_LIB -DQT_QUICKWIDGETS_LIB -DQT_QUICK_LIB -DQT_SERIALPORT_LIB -DQT_SQL_LIB -DQT_SVG_LIB -DQT_UITOOLS_LIB -DQT_USE_QSTRINGBUILDER -DQT_WEBKITWIDGETS_LIB -DQT_WEBKIT_LIB -DQT_WIDGETS_LIB -DQT_XML_LIB -DSIP_VERSION=0x06070b -DWITH_QML -DWITH_QTWEBKIT -D_LIBCPP_ENABLE_CXX17_REMOVED_AUTO_PTR -Dqgis_gui_EXPORTS -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/build -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/actions -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/annotations -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/attributeformconfig -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/symbology -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/attributetable -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/auth -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/callouts -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/codeeditors -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/devtools -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/editorwidgets -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/editorwidgets/core -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/effects -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/elevation -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/history -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/labeling -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/layertree -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/layout -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/locator -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/maptools -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/mesh -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/numericformats -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/ogr -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/plot -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/processing -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/processing/models -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/QGIS-3_32_2/src/gui/providers -I/opt/local/var/
:info:build dyld[94584]: Symbol not found: __ZN7leveldb4port5Mutex4LockEv
:info:build   Referenced from: <26D6445F-3C3D-358C-9876-28E0CA20ECE3> /opt/local/libexec/qt5/lib/QtWebKit.framework/Versions/5/QtWebKit
:info:build   Expected in:     <AA2E5445-8AA3-3EB9-AA5B-2A87085AE255> /opt/local/lib/libleveldb.1.23.0.dylib
:info:build /bin/sh: line 1: 94584 Abort trap: 6           ../../output/bin/crssync
:info:build make[2]: *** [src/crssync/CMakeFiles/synccrsdb] Error 134
:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/build'
:info:build make[1]: *** [src/crssync/CMakeFiles/synccrsdb.dir/all] Error 2
:info:build make[1]: *** Waiting for unfinished jobs....Mutex4LockEv
:info:build   Referenced from: <26D6445F-3C3D-358C-9876-28E0CA20ECE3> /opt/local/libexec/qt5/lib/QtWebKit.framework/Versions/5/QtWebKit
:info:build   Expected in:     <AA2E5445-8AA3-3EB9-AA5B-2A87085AE255> /opt/local/lib/libleveldb.1.23.0.dylib
:info:build /bin/sh: line 1: 94584 Abort trap: 6           ../../output/bin/crssync
:info:build make[2]: *** [src/crssync/CMakeFiles/synccrsdb] Error 134
:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_gis_qgis3/qgis3/work/build'
:info:build make[1]: *** [src/crssync/CMakeFiles/synccrsdb.dir/all] Error 2
:info:build make[1]: *** Waiting for unfinished jobs....

Another issue caused by the leveldb update is that because leveldb conflicts with gperftools, and qgis3 depends on leveldb, there is a conflict between qgis3 and grpc (which requires gperftools).

I tried rebuilding leveldb. This was successful, but building qgis failed the same way.

The error seems to come from linking in qt5-qtwebkit. I tried rebuilding that and it failed with:

:info:build /Library/Developer/CommandLineTools/usr/bin/clang++ -c -pipe -stdlib=libc++ -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_qt5/qt5-qtwebkit/work/qtwebkit-opensource-src-5.9.2/preferredheaders -I/opt/local/include -fno-strict-aliasing -Os -fPIC -std=gnu++11  -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -mmacosx-version-min=13.0 -fvisibility=hidden -fvisibility-inlines-hidden -fno-exceptions -MD -DQT_NO_LINKED_LIST -DLEVELDB_PLATFORM_QT -DOS_MACOSX -D__ASSERT_MACROS_DEFINE_VERSIONS_WITHOUT_UNDERSCORES=0 -DBUILDING_QT__=1 -DNDEBUG -DENABLE_3D_RENDERING=1 -DENABLE_ACCELERATED_2D_CANVAS=1 -DENABLE_BLOB=1 -DENABLE_CANVAS_PATH=1 -DENABLE_CHANNEL_MESSAGING=1 -DENABLE_CSS_BOX_DECORATION_BREAK=1 -DENABLE_CSS_COMPOSITING=1 -DENABLE_CSS_EXCLUSIONS=1 -DENABLE_CSS_FILTERS=1 -DENABLE_CSS_IMAGE_SET=1 -DENABLE_CSS_REGIONS=1 -DENABLE_CSS_SHAPES=1 -DENABLE_CSS_STICKY_POSITION=1 -DENABLE_CSS_TRANSFORMS_ANIMATIONS_UNPREFIXED=1 -DENABLE_DATALIST_ELEMENT=1 -DENABLE_DETAILS_ELEMENT=1 -DENABLE_DEVICE_ORIENTATION=1 -DENABLE_DOWNLOAD_ATTRIBUTE=1 -DENABLE_FAST_MOBILE_SCROLLING=1 -DENABLE_FILTERS=1 -DENABLE_FTPDIR=1 -DENABLE_FULLSCREEN_API=1 -DENABLE_GEOLOCATION=1 -DENABLE_GESTURE_EVENTS=1 -DENABLE_ICONDATABASE=1 -DENABLE_IFRAME_SEAMLESS=1 -DENABLE_INDEXED_DATABASE=1 -DENABLE_INPUT_TYPE_COLOR=1 -DENABLE_INSPECTOR=1 -DENABLE_INSPECTOR_SERVER=1 -DENABLE_JAVASCRIPT_DEBUGGER=1 -DENABLE_LEGACY_NOTIFICATIONS=1 -DENABLE_LEGACY_VIEWPORT_ADAPTION=1 -DENABLE_LEGACY_VENDOR_PREFIXES=1 -DENABLE_LEGACY_WEB_AUDIO=1 -DENABLE_LINK_PREFETCH=1 -DENABLE_METER_ELEMENT=1 -DENABLE_MHTML=1 -DENABLE_NOTIFICATIONS=1 -DENABLE_ORIENTATION_EVENTS=1 -DENABLE_PAGE_VISIBILITY_API=1 -DENABLE_PROGRESS_ELEMENT=1 -DENABLE_RESOLUTION_MEDIA_QUERY=1 -DENABLE_REQUEST_ANIMATION_FRAME=1 -DENABLE_SHARED_WORKERS=1 -DENABLE_SMOOTH_SCROLLING=1 -DENABLE_SQL_DATABASE=1 -DENABLE_SUBPIXEL_LAYOUT=1 -DENABLE_SVG=1 -DENABLE_SVG_FONTS=1 -DENABLE_TOUCH_ADJUSTMENT=1 -DENABLE_TOUCH_EVENTS=1 -DENABLE_TOUCH_SLIDER=1 -DENABLE_VIDEO_TRACK=1 -DENABLE_VIEW_MODE_CSS_MEDIA=1 -DENABLE_WEB_SOCKETS=1 -DENABLE_WEB_TIMING=1 -DENABLE_WORKERS=1 -DENABLE_XHR_TIMEOUT=1 -DWTF_USE_TILED_BACKING_STORE=1 -DWTF_USE_CROSS_PLATFORM_CONTEXT_MENUS=1 -DHAVE_QTQUICK=1 -DHAVE_QTPRINTSUPPORT=1 -DHAVE_QSTYLE=1 -DHAVE_QTTESTLIB=1 -DHAVE_QTPOSITIONING=1 -DHAVE_QTSENSORS=1 -DWTF_USE_LIBXML2=1 -DENABLE_XSLT=1 -DWTF_USE_ZLIB=1 -DWTF_USE_WEBP=1 -DWTF_USE_LIBJPEG=1 -DWTF_USE_LIBPNG=1 -DENABLE_NETSCAPE_PLUGIN_API=1 -DPLUGIN_ARCHITECTURE_UNSUPPORTED=1 -DWTF_USE_3D_GRAPHICS=1 -DENABLE_WEBGL=1 -DENABLE_VIDEO=1 -DWTF_USE_QT_MULTIMEDIA=1 -DHAVE_SQLITE3=1 -DWTF_USE_GRAPHICS_SURFACE=1 -DWTF_USE_LEVELDB=1 -DENABLE_BATTERY_STATUS=0 -DENABLE_CANVAS_PROXY=0 -DENABLE_CSP_NEXT=0 -DENABLE_CSS_GRID_LAYOUT=0 -DENABLE_CSS_HIERARCHIES=0 -DENABLE_CSS_IMAGE_ORIENTATION=0 -DENABLE_CSS_IMAGE_RESOLUTION=0 -DENABLE_CSS_SHADERS=0 -DENABLE_CSS_VARIABLES=0 -DENABLE_CSS3_CONDITIONAL_RULES=0 -DENABLE_CSS3_TEXT=0 -DENABLE_CSS3_TEXT_LINE_BREAK=0 -DENABLE_DASHBOARD_SUPPORT=0 -DENABLE_DATAGRID=0 -DENABLE_DATA_TRANSFER_ITEMS=0 -DENABLE_DIRECTORY_UPLOAD=0 -DENABLE_FILE_SYSTEM=0 -DENABLE_FONT_LOAD_EVENTS=0 -DENABLE_GAMEPAD=0 -DENABLE_HIGH_DPI_CANVAS=0 -DENABLE_INPUT_SPEECH=0 -DENABLE_INPUT_TYPE_DATE=0 -DENABLE_INPUT_TYPE_DATETIME_INCOMPLETE=0 -DENABLE_INPUT_TYPE_DATETIMELOCAL=0 -DENABLE_INPUT_TYPE_MONTH=0 -DENABLE_INPUT_TYPE_TIME=0 -DENABLE_INPUT_TYPE_WEEK=0 -DENABLE_LEGACY_CSS_VENDOR_PREFIXES=0 -DENABLE_MATHML=0 -DENABLE_MEDIA_SOURCE=0 -DENABLE_MEDIA_STATISTICS=0 -DENABLE_MEDIA_STREAM=0 -DENABLE_MICRODATA=0 -DENABLE_MOUSE_CURSOR_SCALE=0 -DENABLE_NAVIGATOR_CONTENT_UTILS=0 -DENABLE_NETWORK_INFO=0 -DENABLE_NOSNIFF=0 -DENABLE_PROXIMITY_EVENTS=0 -DENABLE_QUOTA=0 -DENABLE_RESOURCE_TIMING=0 -DENABLE_SCRIPTED_SPEECH=0 -DENABLE_SECCOMP_FILTERS=0 -DENABLE_SHADOW_DOM=0 -DENABLE_STYLE_SCOPED=0 -DENABLE_TEMPLATE_ELEMENT=0 -DENABLE_TEXT_AUTOSIZING=0 -DENABLE_THREADED_HTML_PARSER=0 -DENABLE_TOUCH_ICON_LOADING=0 -DENABLE_USER_TIMING=0 -DENABLE_VIBRATION=0 -DENABLE_WEB_AUDIO=0 -DBUILDING_leveldb -DBUILDING_WEBKIT -DQT_ASCII_CAST_WARNINGS -DQT_NO_EXCEPTIONS -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -DQT_NO_DEBUG -DQT_GUI_LIB -DQT_CORE_LIB -I. -Iinclude -I. -I../../../Source -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_aqua_qt5/qt5-qtwebkit/work/qtwebkit-opensource-src-5.9.2/Source/include -I/opt/local/libexec/qt5/lib/QtGui.framework/Headers -I/opt/local/libexec/qt5/lib/QtCore.framework/Headers -I.moc -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/OpenGL.framework/Headers -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/AGL.framework/Headers -I/opt/local/libexec/qt5/mkspecs/macx-clang -F/opt/local/libexec/qt5/lib -o .obj/db/repair.o db/repair.cc
:info:build db/c.cc:361:6: error: conflicting types for 'leveldb_writebatch_iterate'
:info:build void leveldb_writebatch_iterate(
:info:build      ^
:info:build /opt/local/include/leveldb/c.h:158:21: note: previous declaration is here
:info:build LEVELDB_EXPORT void leveldb_writebatch_iterate(
:info:build                     ^
:info:build 1 error generated.
:info:build make[2]: *** [.obj/db/c.o] Error 1
:info:build make[2]: *** Waiting for unfinished jobs....

It looks as if qt5-qtwebkit is not detecting leveldb and is using a bundled version, which causes a conflict with the installed header file from the leveldb port. This problem was reported in #65832. The same issue occurs when I tried to build qt5-qtwebkit:

:info:configure Running configure tests...
:info:configure Checking for fontconfig... yes
:info:configure Checking for gccdepends... yes
:info:configure Checking for glx... no
:info:configure Checking for gnuld... no
:info:configure Checking for icu... yes
:info:configure Checking for leveldb... no
:info:configure Checking for libsqlite3... yes
:info:configure Checking for libwebp... yes
:info:configure Checking for libXcomposite... no
:info:configure Checking for libxml2... yes
:info:configure Checking for libXrender... no
:info:configure Checking for libxslt... yes
:info:configure Checking for libzlib... yes

[catap (Kirill A. Korinsky)]

Another issue caused by the leveldb update is that because leveldb conflicts with gperftools, and qgis3 depends on leveldb, there is a conflict between qgis3 and grpc (which requires gperftools).

This is funny but the root cause of that variant and conflicts is qgis3 :) See: ​https://trac.macports.org/ticket/56164

If it irrelevant anymore, I'd happy to removed that hack.

It looks as if qt5-qtwebkit is not detecting leveldb and is using a bundled version, which causes a conflict with the installed header file from the leveldb port. This problem was reported in #65832. The same issue occurs when I tried to build qt5-qtwebkit

On my system it fails because:

> leveldb.cpp:22:10: fatal error: 'helpers/memenv/memenv.h' file not found
> #include <helpers/memenv/memenv.h>
>          ^~~~~~~~~~~~~~~~~~~~~~~~~
> 1 error generated.
> make: *** [obj/leveldb.o] Error 1

which means that qtwebkit uses some internal API of leveldb which exists in helpers. Quite wired.

Anyway, I've prepared a kind of fix: ​https://github.com/catap/macports-ports/tree/leveldb

It contains two commits / changes:

1. a new variant at LevelDB which exposes helpers/memenv/memenv.h;
2. a hack inside qtwebkit to stop using libmemenv which doesn't exist anymore.

May I ask you to give it a try? Thanks.

[catap (Kirill A. Korinsky)]

After a coffee I understand that I can make things much cleaner and simpler.

No need to touch qt ports at all :)

So, I've opened a PR: ​https://github.com/macports/macports-ports/pull/20338 which adds one more hack for LevelDB to expose private API helpers/memenv/memenv.h and creates fake library libmemenv.dylib by default.

It allows to achieve an output of port -v configure qt5-qtwebkit:

Checking for fontconfig... yes
Checking for gccdepends... yes
Checking for glx... no
Checking for gnuld... no
Checking for icu... yes
Checking for leveldb... yes
Checking for libsqlite3... yes
Checking for libwebp... yes
Checking for libXcomposite... no
Checking for libxml2... yes
Checking for libXrender... no
Checking for libxslt... yes
Checking for libzlib... yes

The only issue which should be left is conflict with gperftools. It can be avoided and LevelDB can be switched to tmalloc unconditionally via such dependencies but it had been unstable and crashes at qgis3 port. Anyway, crash may appears by mixing different version of tmalloc because old version links everything statically.

So, here I see two ways:

1. Build leveldb with +tmalloc and confirms that qgis3 doesn't crash;
2. Try to make dependency from gperftools optional.

I have no idea how to test (1), perhabs you may help with this. But I will investigate possibility of (2).

[catap (Kirill A. Korinsky)]

Well. the only reason to have dependency to gperftools from grpc is building benchmarks (!); it can be easy avoided by introducing test variant: ​https://github.com/macports/macports-ports/pull/20339

[catap (Kirill A. Korinsky)]

Long story short: after two PRs ​https://github.com/macports/macports-ports/pull/20338 and ​https://github.com/macports/macports-ports/pull/20339 are merged the issue should be also fixed.

[nilason (Nicklas Larsson)]

Replying to catap:

Long story short: after two PRs ​https://github.com/macports/macports-ports/pull/20338 and ​https://github.com/macports/macports-ports/pull/20339 are merged the issue should be also fixed.

Kirill, thanks for pursuing this! I can confirm ​https://github.com/macports/macports-ports/pull/20338 AND rebuild of qt5-qtwebkit get QGIS working again. I believe you should (and should have) rev-bump the leveldb dependants to force them to rebuild.

[gwright83]

I can confirm that using the patched leveldb and rebuilding qt5-qtwebkit, qgis3 builds successfully. Thank you.

[catap (Kirill A. Korinsky)]

In 47ee39e84e7595d81620e3708aab8446f1dc7b37/macports-ports (master):

leveldb: expose memenv

See: #68132

[catap (Kirill A. Korinsky)]

In 79a9029edbdfadde535c9f2411c69118a159e337/macports-ports (master):

grpc: add test variant

The only reason to have gperftools depedency is to run tests,
which can be avoided by introducing test variant.

See: #68132

[mascguy (Christopher Nielsen)]

Replying to gwright83:

I can confirm that using the patched leveldb and rebuilding qt5-qtwebkit, qgis3 builds successfully.

Since both PRs are now merged, closing as fixed. Thank you for confirming!