Opened 2 years ago

Last modified 19 months ago

#65799 assigned defect

mupdf @1.20.3: builds fail across-the-board: expected ';' after top level declarator

Reported by: mascguy (Christopher Nielsen) Owned by: essandess (Steve Smith)
Priority: Normal Milestone:
Component: ports Version: 2.7.2
Keywords: Cc: mopihopi, gctwnl, harciga, ryandesign (Ryan Carsten Schmidt)
Port: mupdf

Description

Details:

generated/resources/fonts/han/SourceHanSerif-Regular.ttc.c:597528:67:
error: expected ';' after top level declarator
"\x1c\x5e\x7c\x0a\x47\xb4\x18\x40\x49\x23\x55\x2c\x69\x08\x9b\xfb"
                                                                  ^
                                                                  ;

https://ports.macports.org/port/mupdf/details/

Attachments (1)

main.log.xz (18.9 KB) - added by essandess (Steve Smith) 2 years ago.

Download all attachments as: .zip

Change History (24)

comment:1 Changed 2 years ago by mascguy (Christopher Nielsen)

Per the following bug report, Clang is most likely running out of memory:

https://bugs.ghostscript.com/show_bug.cgi?id=699317

The suggested fix is to disable optimization when compiling generated resources. And given how large these files are, that approach sounds appropriate.

comment:2 in reply to:  1 Changed 2 years ago by mascguy (Christopher Nielsen)

Replying to mascguy:

Per the following bug report, Clang is most likely running out of memory:

https://bugs.ghostscript.com/show_bug.cgi?id=699317

The suggested fix is to disable optimization when compiling generated resources. And given how large these files are, that approach sounds appropriate.

Sadly, disabling optimization via -O0 doesn't help: It almost looks as if we might be hitting some type of internal Clang limit.

I've also tried switching from Xcode Clang (version 12 on my 10.15 install), to MacPorts Clang 14 - but no love.

Toolchain folks, apart from tweaking the resources - which are essentially monstrous C files, with binary data encoded in strings - is there any way to increase Clang limits? Or do we need to consider other options, like compiling these via GCC?

Still digging through the project, to get a better idea of what the flow is. And perhaps there's some option to split files. But no luck so far.

Thoughts/suggestions appreciated!

comment:3 Changed 2 years ago by jmroot (Joshua Root)

Some of the build logs have clang segfaulting or reporting an internal error, and those are certainly a bug in clang. Maybe it's related to memory pressure, or maybe not. But with the latest Xcode clang on macOS 12, the error is as in the ticket description, and that doesn't look like the compiler hitting a limit, but just invalid code.

comment:4 in reply to:  3 Changed 2 years ago by essandess (Steve Smith)

Replying to jmroot:

Some of the build logs have clang segfaulting or reporting an internal error, and those are certainly a bug in clang. Maybe it's related to memory pressure, or maybe not. But with the latest Xcode clang on macOS 12, the error is as in the ticket description, and that doesn't look like the compiler hitting a limit, but just invalid code.

The C code is created at build with the script scripts/hexdump.sh. It appears to be valid C code to me, with a terminating ;, but does have over 1.5 million lines of font hex strings. This must be the issue.

wc -l /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_graphics_mupdf/mupdf/work/mupdf-1.20.3-source//generated/resources/fonts/han/SourceHanSerif-Regular.ttc.c
 1550676 /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_graphics_mupdf/mupdf/work/mupdf-1.20.3-source//generated/resources/fonts/han/SourceHanSerif-Regular.ttc.c

SourceHanSerif-Regular.ttc.c:

// This is an automatically generated file. Do not edit.
const unsigned char _binary_SourceHanSerif_Regular_ttc[] =
"\x74\x74\x63\x66\x00\x01\x00\x00\x00\x00\x00\x04\x00\x00\x00\x1c"
"\x00\x00\x01\x18\x00\x00\x02\x14\x00\x00\x03\x10\x4f\x54\x54\x4f"
…
"\x03\x70\x03\x70\x03\x70\x03\x70\x03\x70\x03\x70\x03\x70\x03\x70"
"\x03\x70\x03\x70"
""
;
const unsigned int _binary_SourceHanSerif_Regular_ttc_size = sizeof(_binary_SourceHanSerif_Regular_ttc) - 1;

Changed 2 years ago by essandess (Steve Smith)

Attachment: main.log.xz added

comment:5 Changed 2 years ago by essandess (Steve Smith)

FWIW, here's a successful build log on an x86_64 box where this works. I also have another x86_64 box where clang segfaults. Exact same macOS and Xcode versions and builds.

I've tried to compare differences between /etc/sysctl.conf and sudo launchctl limits and don't see any differences there.

comment:6 Changed 2 years ago by essandess (Steve Smith)

This is unrelated. New ticket: #65808

More troubleshooting references. Using MacPorts emacs on the C file also segfaults (!!) Using vi works. This is only a 113 MB file, so there's no way that should happen. Related issue?

Last edited 2 years ago by essandess (Steve Smith) (previous) (diff)

comment:7 Changed 2 years ago by mopihopi

Cc: mopihopi added

comment:8 Changed 2 years ago by essandess (Steve Smith)

Still an issue on one box after recent macOS/Xcode updates:

macOS 12.6 21G115 x86_64
Xcode 14.0 14A309

comment:9 Changed 2 years ago by mascguy (Christopher Nielsen)

Worst-case, are there any libs/utilities out there, to embed an external binary blob in an object file?

Alternatively, does this project support dynamically loading such files into memory at runtime, to sidestep this? (While I haven't looked at the code yet, it's not clear why this approach is necessary...)

comment:10 Changed 2 years ago by essandess (Steve Smith)

It’s weird. It compiles successfully on a 128 GB Mac Pro, but fails on a 64 GB Mac Mini, exact same configuration otherwise, and I’ve sanity checked launchctl limits and specific sysctl settings.

I’m speculating on the side of a clang bug.

comment:11 Changed 2 years ago by gctwnl

Is there anything being done about this? Might we for instance as a workaround exclude this font in the config? And not only this font but also others, as mine now fails (clang crash) on NotoSansOldPersian-Regular.otf. Maybe exclude the Noto fonts.

Last edited 2 years ago by gctwnl (previous) (diff)

comment:12 Changed 2 years ago by gctwnl

From StackExchange:

One possible solution to this issue is to tell the mupdf configure script not to compile the NotoSansOldPersian-Regular.otf font. You can do this by passing the --without-font-truetype option to the ./configure script when building mupdf. This will prevent mupdf from trying to compile the font, which should allow the compilation process to complete successfully.

comment:13 Changed 2 years ago by jmroot (Joshua Root)

Sounds like a reasonable workaround for the short term. Has anyone opened an issue with LLVM?

comment:14 Changed 20 months ago by gctwnl

I am returning to this issue (I really need mupdf and I can't upgrade to Ventura in case that would fix anything)

I am trying to see what I have to change, but I'm getting lost. I have no idea how to implement "You can do this by passing the --without-font-truetype option to the ./configure script when building mupdf." if only because use_configure is set to no and I see only a single configure script in the entire distribution.

Last edited 20 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:15 Changed 20 months ago by ryandesign (Ryan Carsten Schmidt)

Cc: gctwnl harciga ryandesign added

Yeah I don't understand that either. This project does not have a configure script. (It includes bundled copies of other libraries, some of which have configure scripts, but that's irrelevant.) Maybe some Linux package manager has added a configure script for mupdf, and it has this option.


I was able to overcome the clang crash with 1.20.3 by disabling parallel building; try this:

sudo port install mupdf build.jobs=1

I am also preparing an update to 1.21.1 in which I don't see the clang crash even with parallel building.

comment:16 Changed 20 months ago by gctwnl

Indeed, disabling parallel builds is a workaround. Great!

comment:17 Changed 20 months ago by ryandesign (Ryan Carsten Schmidt)

In 8c3eac6324ce4cdec2d1389d2353731edc285b02/macports-ports (master):

mupdf: Disable parallel build

See: #65799

comment:18 Changed 19 months ago by markmentovai (Mark Mentovai)

Given that 8c3eac6324ce (2023-02-26) disabled parallel building, is this now considered fixed?

Comment 15 also says that even without parallel building, mupdf 1.21.1 doesn’t have this problem. 9cd1afbaa7d2 (2023-04-04) updated mupdf to 1.21.1. Is it advisable to revert 8c3eac6324ce now?

Last edited 19 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:19 Changed 19 months ago by ryandesign (Ryan Carsten Schmidt)

Yes, my intention was to remove the parallel building restriction and mark this ticket resolved when updating to 1.21.1 but you beat me to it with the update. I don't remember what else I was still working on with my 1.21.1 update or why I hadn't committed it already... I'll go check.

comment:20 Changed 19 months ago by ryandesign (Ryan Carsten Schmidt)

Looks like I was making extensive modifications to the port. mupdf comes bundled with many libraries that we already have in MacPorts and I was trying to make the port use the existing MacPorts copies instead of its bundled ones.

comment:21 Changed 19 months ago by markmentovai (Mark Mentovai)

Is that work in progress something you can share so we can iterate collaboratively?

comment:22 in reply to:  19 Changed 19 months ago by mascguy (Christopher Nielsen)

Replying to ryandesign:

Yes, my intention was to remove the parallel building restriction and mark this ticket resolved when updating to 1.21.1 but you beat me to it with the update. I don't remember what else I was still working on with my 1.21.1 update or why I hadn't committed it already... I'll go check.

Just for giggles, I tried re-enabling parallel builds for v1.21.1, and still see the failure locally. (At least on 10.13, with Xcode 9.4.1.)

So unless I'm missing something, it looks like it's still problematic in some circumstances.

comment:23 Changed 19 months ago by gctwnl

Given the extreme size of code statements (hard coded huge data elements) and the proven instability with some compilers, disabling parallel builds may be a good default for this one.

Note: See TracTickets for help on using tickets.