Opened 3 years ago

Last modified 2 years ago

#55385 reopened defect

ocaml: fails to build on 10.6 and lower

Reported by: mojca (Mojca Miklavec) Owned by: pmetzger (Perry E. Metzger)
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc:
Port: ocaml

Description (last modified by mojca (Mojca Miklavec))

OCaml port fails to build on 10.6 and lower. It might help to use a newer compiler, but it makes sense to test the hypothesis before blindly adding items to compilers blacklist.

/usr/bin/gcc-4.2 -c -DSYS_macosx -DMODEL_default -o amd64.o amd64.S || \
	{ echo "If your assembler produced syntax errors, it is probably";\
          echo "unhappy with the preprocessor. Check your assembler, or";\
          echo "try producing amd64.o by hand.";\
          exit 2; }
amd64.S:708:Unknown pseudo-op: .string
amd64.S:708:Rest of line ignored. 1st junk character valued 34 (").
If your assembler produced syntax errors, it is probably
unhappy with the preprocessor. Check your assembler, or
try producing amd64.o by hand.

If it turns out that it in fact cannot be built, the port should at least say so. But I guess it's just an issue with too old compiler.

On 10.5 it failed with

/usr/bin/gcc-4.2 -bundle -flat_namespace -undefined suppress                    -Wl,-no_compact_unwind -o libcamlrun_shared.so interp.pic.o misc.pic.o stacks.pic.o fix_code.pic.o startup_aux.pic.o startup.pic.o freelist.pic.o major_gc.pic.o minor_gc.pic.o memory.pic.o alloc.pic.o roots.pic.o globroots.pic.o fail.pic.o signals.pic.o signals_byt.pic.o printexc.pic.o backtrace_prim.pic.o backtrace.pic.o compare.pic.o ints.pic.o floats.pic.o str.pic.o array.pic.o io.pic.o extern.pic.o intern.pic.o hash.pic.o sys.pic.o meta.pic.o parsing.pic.o gc_ctrl.pic.o terminfo.pic.o md5.pic.o obj.pic.o lexing.pic.o callback.pic.o debugger.pic.o weak.pic.o compact.pic.o finalise.pic.o custom.pic.o dynlink.pic.o spacetime.pic.o afl.pic.o unix.pic.o main.pic.o -lcurses -lpthread                  
ld: unknown option: -no_compact_unwind

Attachments (1)

ocaml.ppc.gcc6.fail.log (263.3 KB) - added by kencu (Ken) 3 years ago.
ocaml 4.05 build failure on PPC with gcc6

Download all attachments as: .zip

Change History (48)

comment:1 Changed 3 years ago by mojca (Mojca Miklavec)

Description: modified (diff)

comment:2 Changed 3 years ago by mojca (Mojca Miklavec)

Description: modified (diff)

comment:3 Changed 3 years ago by mojca (Mojca Miklavec)

Description: modified (diff)

comment:4 Changed 3 years ago by mojca (Mojca Miklavec)

Another totally weird failure to build coq.

comment:5 Changed 3 years ago by pmetzger (Perry E. Metzger)

My guess is that indeed we need to depend (on those older platforms only) on a newer C compiler.

I lack direct access to a 10.6 or 10.5 machine, so it may be hard for me to do tests without blind changes.

How many users do those old versions have btw? I presume it is considered important to fix them?

For the 10.10 failure, am I reading the log correctly? Did something kill the build with -9? Maybe it isn't a real build failure but was rather a one time oddity of some sort?

comment:6 Changed 3 years ago by mojca (Mojca Miklavec)

I have no numbers, but 10.6 is still somehow popular for enthusiasts. Maintainers are not expected to fix the problems if they don't want to, it purely depends on the enthusiasm of each individual. If it's just a matter of blacklisting gcc 4.2, it might be worth a shot. If it requires heavy patching, there's no need to waste your precious time.

For coq on 10.10 check this build (once ready).

Last edited 3 years ago by mojca (Mojca Miklavec) (previous) (diff)

comment:7 Changed 3 years ago by kencu (Ken)

I can check easily. I'll get back to you.

10.6 is a "sticky" system version, with adherents like me. There are lots of reasons -- Apple just hit a sweet spot for many of us with that version.

comment:8 Changed 3 years ago by pmetzger (Perry E. Metzger)

Please do check if it can be fixed just by altering the compiler. A patch for the Portfile that's conditioned on OS X <= 10.6 would be appreciated too.

comment:9 Changed 3 years ago by kencu (Ken)

My default 10.6.8 system is set up with LibcxxOnOlderSystems, using libc++, and the default compiler is set to clang-3.9. On that system (which I recommend for everyone to use, and dearly wish we had a buildbot for), ocaml built straight through without touching anything:

$ port -v installed ocaml
The following ports are currently installed:
  ocaml @4.02.2_1 (active) platform='darwin 10' archs='x86_64' date='2017-11-22T09:22:43-0800'

I will explore exactly what works, but in my mind, you should either let the default compiler do it, or jump directly to a new clang (like 4 or 5). I personally think it's kind of tedious to spend too much time finding the oldest compiler that works (a moving target for the next version of the port anyway, usually).

comment:10 Changed 3 years ago by mojca (Mojca Miklavec)

Repeated attempt of build coq was successful.

Let's give blacklisting gcc 4.0 and 4.2 one shot. If that doesn't help, we can figure out the details once again.

comment:11 Changed 3 years ago by mojca (Mojca Miklavec)

In ccfdeabbecfe8d97d1e888eefea91b4c85f0525b/macports-ports:

ocaml: blacklist gcc 4.0 and 4.2

See: #55385

comment:12 Changed 3 years ago by mojca (Mojca Miklavec)

OK, for 10.5 this was a failed experiment. The problem existed before and doesn't seem to be connected with the compiler in any way (sorry, I'll help you remove the blacklisting :). See the existing ticket #40956 with claims that the problem was solved after patching the Makefiles.

10.6/i386 using clang fails with

ld: absolute addressing (perhaps -mdynamic-no-pic) used in _caml_system__code_begin from i386.pic.o not allowed in slidable image. Use '-read_only_relocs suppress' to enable text relocs
clang: error: linker command failed with exit code 1 (use -v to see invocation)

10.6/i386 using clang-3.4 fails with

ld: illegal text-relocoation (direct reference) to (global,weak) _caml_gc_regs in roots.pic.o from _caml_call_gc in i386.pic.o for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

10.6/x86_64 using clang 10.6/x86_64 using clang-3.4

Last edited 3 years ago by mojca (Mojca Miklavec) (previous) (diff)

comment:13 Changed 3 years ago by mojca (Mojca Miklavec)

I'm stealing this ticket for a while for other issues, but my suggestion would be the following:

  • In case it's possible to trivially fix this particular issue first, that would be helpful because of the planned changes on 100+ other ports. (Btw: where did you get the number 300? Did you also count the patchfiles?) That is: if you move the other ports before fixing this, all those changes will generate errors on the buildbot just because of the failure to build dependencies and you would need to rebuild them later. But don't spend too much time on it as dealing with old systems is not the highest priority, it makes more sense to first make sure that ocaml works well on newer systems. (If you could remove that flag by some conditional patching on < 10.6, fine, else don't bother.)
  • You may go through some old tickets (some are 10 years old) where the port field contains ocaml and figure out if any of them happen to be addressed already, or if an upgrade to the latest version solves the problem. If that requires small independent patches, you can submit them separately, so that we apply them before the "big move". If ports are obsolete, feel free to delete them. If some issues have been addressed, ask on IRC if someone can close them for you (or collect such requests in another ticket perhaps?)
  • I would suggest to then try to move ports to a different subfolder and add an additional category. And let the buildbots try to build all of them. Just moving the ports should be sufficient to trigger a rebuild. You may collect some basic statistics about which ports need an updated etc. along the way.
  • Then collect statistics about which ports failed, so that you can address those later.

Just a few ideas.

comment:14 Changed 3 years ago by pmetzger (Perry E. Metzger)

So just to be clear, what would everyone recommend we do at this point? (I'd like to get moving with further downstream changes.)

Last edited 3 years ago by pmetzger (Perry E. Metzger) (previous) (diff)

comment:15 Changed 3 years ago by mojca (Mojca Miklavec)

Just don't get swamped into solving this particular issue. I'll try another blacklisting for 10.6 and if that doesn't work, I'll leave it to others to figure it out.

My suggestion would be to either look at some existing tickets (if they don't require too much extra effort) or else concentrate into upgrading/fixing some highest priority ocaml-based ports, or perhaps start by moving the ports & adding the category with some assistance of scripting. Just do whatever will make you most productive and submit pull request(s).

If you don't plan to move the ports just yet, you can also send me a list of ports, so that we can quickly check on the buildbot which ones are broken. If you do plan to move them, that could be one of the first steps that will get you the builds as a side product.

Last edited 3 years ago by mojca (Mojca Miklavec) (previous) (diff)

comment:16 Changed 3 years ago by mojca (Mojca Miklavec)

In 20caa81a96ce015ff8b1be8bff27f1b17532ba89/macports-ports:

ocaml: blacklist clang from 10.6

See: #55385

comment:17 Changed 3 years ago by kencu (Ken)

-Wl,-read_only_relocs,suppress should probably be in base for all PPC and i386 builds. It is often needed. I think it used to be in base once upon a time.

comment:18 Changed 3 years ago by pmetzger (Perry E. Metzger)

kencu, could you supply a patch after verifying that it works?

comment:19 Changed 3 years ago by kencu (Ken)

will do. I guess you guys are in a bit of a hurry to get this going. Sorry I'm somewhat tied up in my day job just now.

comment:20 Changed 3 years ago by pmetzger (Perry E. Metzger)

I have a huge number of OCaml ports to deal with so getting the number of open problems down is a priority. :(

I would try fixing 10.5 and 10.6 myself but I have no machines I can test on.

Last edited 3 years ago by pmetzger (Perry E. Metzger) (previous) (diff)

comment:21 Changed 3 years ago by kencu (Ken)

I am working on this right now. I'm sorry I made an error above, and I built a slightly older ocaml version above. The current version that we're interested in, 4.0.5 fails to build due to a missing strnlen function.

../boot/ocamlrun ../ocamlc -nostdlib -I ../stdlib -linkall -I ../otherlibs/unix -o ocamldebug -linkall ../otherlibs/unix/unix.cma ../utils/config.cmo ../utils/tbl.cmo ../utils/misc.cmo ../utils/identifiable.cmo ../utils/numbers.cmo ../utils/arg_helper.cmo ../utils/clflags.cmo ../utils/consistbl.cmo ../utils/warnings.cmo ../utils/terminfo.cmo ../parsing/location.cmo ../parsing/longident.cmo ../parsing/docstrings.cmo ../parsing/syntaxerr.cmo ../parsing/ast_helper.cmo ../parsing/ast_mapper.cmo ../parsing/ast_iterator.cmo ../parsing/attr_helper.cmo ../parsing/builtin_attributes.cmo ../typing/ident.cmo ../typing/path.cmo ../typing/types.cmo ../typing/btype.cmo ../typing/primitive.cmo ../typing/typedtree.cmo ../typing/subst.cmo ../typing/predef.cmo ../typing/datarepr.cmo ../typing/cmi_format.cmo ../typing/env.cmo ../typing/oprint.cmo ../typing/ctype.cmo ../typing/printtyp.cmo ../typing/mtype.cmo ../typing/envaux.cmo ../bytecomp/runtimedef.cmo ../bytecomp/bytesections.cmo ../bytecomp/dll.cmo ../bytecomp/meta.cmo ../bytecomp/symtable.cmo ../bytecomp/opcodes.cmo ../driver/compdynlink.cmo ../toplevel/genprintval.cmo int64ops.cmo primitives.cmo unix_tools.cmo debugger_config.cmo parameters.cmo lexer.cmo input_handling.cmo question.cmo debugcom.cmo exec.cmo source.cmo pos.cmo checkpoints.cmo events.cmo program_loading.cmo symbols.cmo breakpoints.cmo trap_barrier.cmo history.cmo printval.cmo show_source.cmo time_travel.cmo program_management.cmo frames.cmo eval.cmo show_information.cmo loadprinter.cmo parser.cmo command_line.cmo main.cmo
File "_none_", line 1:
Error: Error on dynamically loaded library: ../otherlibs/unix/dllunix.so: dlopen(../otherlibs/unix/dllunix.so, 138): Symbol not found: _strnlen
  Referenced from: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_ocaml/ocaml/work/ocaml-4.05.0/otherlibs/unix/dllunix.so
  Expected in: flat namespace
 in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_ocaml/ocaml/work/ocaml-4.05.0/otherlibs/unix/dllunix.so
make[3]: *** [ocamldebug] Error 2
make[2]: *** [ocamldebugger] Error 2
make[1]: *** [opt.opt] Error 2
make: *** [world.opt] Error 2
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_ocaml/ocaml/work/ocaml-4.05.0" && unset LD_PREBIND LD_PREBIND_ALLOW_OVERLAP && /usr/bin/make world.opt 

The snowleopard_fixes portgroup might normally fix this, but unfortunately it's somewhere in the ocaml build itself where it fails, not standard c++, so the configure.flags manipulations don't work.

Short story - at present it won't build on 10.6 or less, even with a new compiler. I can work on it, but it will take some time it appears. A patch for the missing function, maybe, or see how to link in the -lsnowleopardfixes library somehow in that link stage.

comment:22 Changed 3 years ago by mojca (Mojca Miklavec)

Perry, for all I can say, don't worry about these issues and concentrate on more important ones for now. If this was a trivial fix with blacklisting a compiler or two, we could do it, but it makes no sense to put too much effort in this compared to the importance of cleaning up the ports for everyone else. You may try to submit a super low priority bug report upstream (but again, don't worry about it).

comment:23 Changed 3 years ago by kencu (Ken)

Does anyone know how to add a link library to an ocaml build? I think all we need to do (AFAIK) is figure out how to get whatever ocaml sees as -lsnowleopardfixes.dylib added onto that build line that fails

../boot/ocamlrun ../ocamlc -nostdlib -I ../stdlib -linkall -I ../otherlibs/unix -o ocamldebug -linkall ../otherlibs/unix/unix.cma ../utils/config.cmo ../utils/tbl.cmo ../utils/misc.cmo ../utils/identifiable.cmo ../utils/numbers.cmo ../utils/arg_helper.cmo ../utils/clflags.cmo ../utils/consistbl.cmo ../utils/warnings.cmo ../utils/terminfo.cmo ../parsing/location.cmo ../parsing/longident.cmo ../parsing/docstrings.cmo ../parsing/syntaxerr.cmo ../parsing/ast_helper.cmo ../parsing/ast_mapper.cmo ../parsing/ast_iterator.cmo ../parsing/attr_helper.cmo ../parsing/builtin_attributes.cmo ../typing/ident.cmo ../typing/path.cmo ../typing/types.cmo ../typing/btype.cmo ../typing/primitive.cmo ../typing/typedtree.cmo ../typing/subst.cmo ../typing/predef.cmo ../typing/datarepr.cmo ../typing/cmi_format.cmo ../typing/env.cmo ../typing/oprint.cmo ../typing/ctype.cmo ../typing/printtyp.cmo ../typing/mtype.cmo ../typing/envaux.cmo ../bytecomp/runtimedef.cmo ../bytecomp/bytesections.cmo ../bytecomp/dll.cmo ../bytecomp/meta.cmo ../bytecomp/symtable.cmo ../bytecomp/opcodes.cmo ../driver/compdynlink.cmo ../toplevel/genprintval.cmo int64ops.cmo primitives.cmo unix_tools.cmo debugger_config.cmo parameters.cmo lexer.cmo input_handling.cmo question.cmo debugcom.cmo exec.cmo source.cmo pos.cmo checkpoints.cmo events.cmo program_loading.cmo symbols.cmo breakpoints.cmo trap_barrier.cmo history.cmo printval.cmo show_source.cmo time_travel.cmo program_management.cmo frames.cmo eval.cmo show_information.cmo loadprinter.cmo parser.cmo command_line.cmo main.cmo

and it should work. Maybe.

I just haven't a clue how to do that in ocaml-speak.

Last edited 3 years ago by kencu (Ken) (previous) (diff)

comment:24 Changed 3 years ago by kencu (Ken)

fixed, at least for 10.6.8

$ port -v installed ocaml
The following ports are currently installed:
  ocaml @4.05.0_0 (active) platform='darwin 10' archs='x86_64' date='2017-12-15T14:13:57-0800'

comment:25 Changed 3 years ago by pmetzger (Perry E. Metzger)

I see you've requested a pull on github. I've approved it, though the merge should wait until the CI system finishes its check.

comment:26 Changed 3 years ago by kencu (Ken)

Resolution: fixed
Status: newclosed

In 61940fe8470441051f7a77e62176e129d1e8e902/macports-ports:

ocaml: fix build error due to missing strnlen

fixes build on 10.6.8 and earlier
closes: #55385

comment:27 Changed 3 years ago by pmetzger (Perry E. Metzger)

Resolution: fixed
Status: closedreopened

comment:28 Changed 3 years ago by pmetzger (Perry E. Metzger)

Reopened because the i386 and ppc builds remain broken for now.

Changed 3 years ago by kencu (Ken)

Attachment: ocaml.ppc.gcc6.fail.log added

ocaml 4.05 build failure on PPC with gcc6

comment:29 Changed 3 years ago by kencu (Ken)

PPC may not work out. Fails with clang-3.4 (of course), but fails with gcc6 as well. Cause of error not obvious to me.

comment:30 Changed 3 years ago by pmetzger (Perry E. Metzger)

Does PPC work with the system compiler? If not, I'm inclined to mark the port only for i386 and x86-64. Thoughts?

comment:31 Changed 3 years ago by kencu (Ken)

Yeah, the PPC build doesn't work with any compiler just yet. Looks like some kind of assembly code issue in the build script that I'm not inclined to bother with at present. If anyone ever cares about it, may help them push on it then. I wonder if any ocaml build ever worked on PPC.

Re: 32bit 10.6.8, I don't have a 10.6.8 i386 setup to test on, unfortunately. I think adding that linker flag I mentioned in the PR would probably fix that build. Only thing I can do is push a fix to the buildbot and see if it fails or not. Have to remember how to test for a 32bit build -- some kind of ${build.arch} test, I think.

comment:32 Changed 3 years ago by pmetzger (Perry E. Metzger)

Try to fix the i386. I'm going to blacklist ppc.

comment:33 Changed 3 years ago by pmetzger (Perry E. Metzger)

So Ken, it seems that ocamlbuild isn't building right on 10.6 because ocaml tries to invoke the same C compiler that it was built with, which isn't the default, and which is a bit hard to specify in port files. Not sure how to proceed from here.

comment:34 Changed 3 years ago by kencu (Ken)

I think I know what you mean. We had the same issue with ghc, and had to fix this kind of problem there too by specifying a dependency on the build compiler. Can you git me a log or something to show how it fails?

comment:35 Changed 3 years ago by pmetzger (Perry E. Metzger)

It's pretty mysterious, but look at:

https://build.macports.org/builders/ports-10.6_i386_legacy-builder/builds/33965/steps/install-dependencies/logs/stdio

Things are going along quite smoothly and suddenly they blow up. I can't quite explain what's going on. The important bit is:

ocamlc.opt -w L -w R -w Z -I src -I +unix -safe-string -bin-annot -c src/ocamlbuild.mli
ocamlc.opt -w L -w R -w Z -I src -I +unix -safe-string -bin-annot -c src/ocamlbuild.ml
ocamlc.opt -I +unix -I src -o ocamlbuild.byte unix.cma src/ocamlbuild_pack.cmo src/ocamlbuild_plugin.cmo src/ocamlbuild_executor.cmo src/ocamlbuild_unix_plugin.cmo src/ocamlbuild.cmo
ocamlc.opt -a -o src/ocamlbuildlib.cma src/ocamlbuild_pack.cmo src/ocamlbuild_plugin.cmo src/ocamlbuild_executor.cmo src/ocamlbuild_unix_plugin.cmo
ocamlopt.opt -for-pack Ocamlbuild_pack -w L -w R -w Z -I src -I +unix -safe-string -bin-annot -c src/const.ml
sh: /opt/local/bin/clang-mp-3.4: No such file or directory
File "src/const.ml", line 1:
Error: Assembler error, input left in file /opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_ocaml_ocaml-ocamlbuild/ocaml-ocamlbuild/work/.tmp/camlasm709cf1.s
make: *** [src/const.cmx] Error 2
make: Leaving directory `/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_ocaml_ocaml-ocamlbuild/ocaml-ocamlbuild/work/ocamlbuild-0.12.0'
Command failed:  cd "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_ocaml_ocaml-ocamlbuild/ocaml-ocamlbuild/work/ocamlbuild-0.12.0" && /usr/bin/make -w all 
Exit code: 2

comment:36 Changed 3 years ago by kencu (Ken)

Yep. Exactly the same error as ghc. This is the relevant part:

sh: /opt/local/bin/clang-mp-3.4: No such file or directory

ocaml will need a runtime dep on macports-clang-3.4 on 10.6.

We might as well do it all at once -- are there any other systems that fail?

Last edited 3 years ago by kencu (Ken) (previous) (diff)

comment:37 Changed 3 years ago by pmetzger (Perry E. Metzger)

Both x86_64 and i386 on 10.6 errored out in the buildbot. I don't know if we support *86 on 10.5, I didn't see any errors from that?

ppc is broken on both 10.5 and 10.6 in the buildbot of course, but we knew that.

comment:38 Changed 3 years ago by kencu (Ken)

10.5 Intel most likely works, so we'll make sure that is included in the test.

The 10.7 buildbot seems to be broken or offline -- I'll test it on my local VM.

10.8+ build OK with the default compiler?

comment:39 Changed 3 years ago by pmetzger (Perry E. Metzger)

I'm pretty sure 10.7+ is okay with the default compiler, not just 10.8.

comment:40 Changed 3 years ago by pmetzger (Perry E. Metzger)

BTW, for those reading this, I finally understood the difference in the transcript. (ocamlc.opt is a bytecode compiler, not a native compiler. ocamlopt.opt is the native compiler. It was dying the first time it invoked the native compiler.) So the failure is not so mysterious after all, I just had a reading comprehension failure.

Last edited 3 years ago by pmetzger (Perry E. Metzger) (previous) (diff)

comment:41 Changed 3 years ago by pmetzger (Perry E. Metzger)

Looks like we have another fail

https://build.macports.org/builders/ports-10.6_i386_legacy-watcher/builds/11338

I can't quite figure out why ocamlbuild isn't building now (too many files to sift through right now, but feel free...)

comment:42 Changed 3 years ago by kencu (Ken)

ocaml-ocalmbuild built on 10.6.8 when I just force-rebuilt it:

<https://build.macports.org/builders/ports-10.6_i386_legacy-builder/builds/34138/steps/install-port/logs/stdio>

It's a big, complicated mess of ports. We'll pick off the errors step by step.

comment:43 Changed 3 years ago by pmetzger (Perry E. Metzger)

The errors in the build run I posted seemed to indicate ocaml-ocamlbuild hadn't rebuilt during that run and was the cause of most of the failures... any idea why the difference?

comment:44 Changed 3 years ago by kencu (Ken)

Timing, probably. When that run you specified was run, the last attempt at ocaml-ocamlbuild had failed, and so the buildbot will not try to build it again unless something is done (either a forced rebuild attempt, or a change in the Portfile pushed to master).

comment:45 Changed 2 years ago by pmetzger (Perry E. Metzger)

kencu, I'm kind of tempted to close this as "will not fix". Thoughts?

comment:46 Changed 2 years ago by kencu (Ken)

Mmm. ocaml builds and runs fine for me on 10.6.8.

$ port -v installed | grep ocaml
  ocaml @4.05.0_0 (active) platform='darwin 10' archs='x86_64' date='2017-12-15T14:13:57-0800'

I think it more a buildbot issue. On the buildbot, ocaml builds with clang-3.4, and thereafter, requires clang-3.4 to run because it's hard-coded into the scripts.

But there was no run dep on clang-3.4, and so if someone installed ocaml and tried to use it, but clang-3.4 is not installed, ocaml failed.

Looks fixable to me. Is it still causing errors for people?

Last edited 2 years ago by kencu (Ken) (previous) (diff)

comment:47 Changed 2 years ago by pmetzger (Perry E. Metzger)

It's certainly causing me headaches when the buildbot complains.

Note: See TracTickets for help on using tickets.