Opened 11 years ago

Closed 11 years ago

#39652 closed defect (fixed)

Base/ports tarballs and PortIndexes are not being updated

Reported by: mndavidoff (Monte Davidoff) Owned by: jmroot (Joshua Root)
Priority: High Milestone:
Component: server/hosting Version: 2.1.3
Keywords: Cc: wsiegrist@…, neverpanic (Clemens Lang), su-v, b.forbort@…, MarcusCalhoun-Lopez (Marcus Calhoun-Lopez), larryv (Lawrence Velázquez), jdgleeson, drkp (Dan Ports), mail@…, nonstop.server@…, skymoo (Adam Mercer), jwhowse4, stromnov (Andrey Stromnov), mojca (Mojca Miklavec), cjones051073 (Chris Jones)
Port:

Description

It appears port sync is not downloading an up-to-date ports.tar. It seems to be stuck downloading a ports.tar that's currently about 20 hours old.

$ date
Sat Jul  6 10:45:29 PDT 2013
$ sudo port sync
--->  Updating the ports tree
$ ls -l /opt/local/var/macports/sources/rsync.macports.org/release/tarballs
total 130176
-rw-r--r--   1 root  admin   9759688 Jul  5 14:31 PortIndex
-rw-r--r--   1 root  admin       512 Jun 16 10:31 PortIndex.rmd160
drwxr-xr-x  24 root  wheel       816 Jan 31 09:30 base
-rw-r--r--   1 root  admin   3594240 Jul  5 15:03 base.tar
-rw-r--r--   1 root  admin       512 Jul  5 15:03 base.tar.rmd160
drwxr-xr-x  51 root  wheel      1734 Jul  6 10:46 ports
-rw-r--r--   1 root  admin  53278720 Jul  5 15:03 ports.tar
-rw-r--r--   1 root  admin       512 Jul  5 15:03 ports.tar.rmd160
$ date
Sat Jul  6 10:46:25 PDT 2013

Change History (36)

comment:1 Changed 11 years ago by cooljeanius (Eric Gallager)

cal might want to be cc-ed on this, as he brought up the same issue on macports-dev: https://lists.macosforge.org/pipermail/macports-dev/2013-July/023223.html

comment:2 Changed 11 years ago by neverpanic (Clemens Lang)

Cc: cal@… added
Component: portsserver/hosting
Owner: changed from macports-tickets@… to wsiegrist@…
Priority: NormalHigh

I was actually just writing a mail for an issue originally reported in IRC. I think this is best assigned to our admins.

comment:3 Changed 11 years ago by su-v

Cc: suv-sf@… added

Cc Me!

comment:4 Changed 11 years ago by larryv (Lawrence Velázquez)

Summary: port sync downloads out-of-date ports.tarrsync tarballs and PortIndexes are not being updated

It’s not just ports.tar.

% TZ=UTC rsync rsync://rsync.macports.org/release/tarballs/
drwxr-xr-x        4096 2012/07/20 20:37:09 .
-rw-r--r--     3594240 2013/07/05 22:03:03 base.tar
-rw-r--r--         512 2013/07/05 22:03:05 base.tar.rmd160
-rw-r--r--    53278720 2013/07/05 22:03:04 ports.tar
-rw-r--r--         512 2013/07/05 22:03:05 ports.tar.rmd160
drwxr-xr-x        4096 2013/07/05 22:02:16 PortIndex_darwin_10_i386
drwxr-xr-x        4096 2013/07/05 22:02:19 PortIndex_darwin_11_i386
drwxr-xr-x        4096 2013/07/05 22:02:21 PortIndex_darwin_12_i386
drwxr-xr-x        4096 2013/07/05 22:02:09 PortIndex_darwin_8_i386
drwxr-xr-x        4096 2013/07/05 22:02:06 PortIndex_darwin_8_powerpc
drwxr-xr-x        4096 2013/07/05 22:02:14 PortIndex_darwin_9_i386
drwxr-xr-x        4096 2013/07/05 22:02:11 PortIndex_darwin_9_powerpc
%

comment:5 Changed 11 years ago by b.forbort@…

Cc: b.forbort@… added

Cc Me!

comment:6 Changed 11 years ago by MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)

Cc: mcalhoun@… added

Cc Me!

comment:7 Changed 11 years ago by larryv (Lawrence Velázquez)

Cc: larryv@… added

Cc Me!

comment:8 Changed 11 years ago by larryv (Lawrence Velázquez)

Summary: rsync tarballs and PortIndexes are not being updatedBase/ports tarballs and PortIndexes are not being updated

As John just noted, the daily tarball is out of date, too. Which makes sense, since it’s generated by the rsync server.

% curl -s https://distfiles.macports.org/ports.tar.gz | TZ=UTC pax -vzT 201307050000 | cut -c 43- | sort -r | head -n 20
Jul  5 22:02 ports/PortIndex_darwin_9_powerpc/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_9_powerpc
Jul  5 22:02 ports/PortIndex_darwin_9_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_9_i386
Jul  5 22:02 ports/PortIndex_darwin_8_powerpc/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_8_powerpc
Jul  5 22:02 ports/PortIndex_darwin_8_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_8_i386
Jul  5 22:02 ports/PortIndex_darwin_12_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_12_i386
Jul  5 22:02 ports/PortIndex_darwin_11_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_11_i386
Jul  5 22:02 ports/PortIndex_darwin_10_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_10_i386
Jul  5 22:02 ports/PortIndex.quick
Jul  5 22:02 ports/PortIndex
Jul  5 21:31 ports/devel/qt4-creator-mac/Portfile
Jul  5 21:31 ports/devel/qt4-creator-mac
Jul  5 21:31 ports/PortIndex_darwin_9_powerpc/PortIndex
Jul  5 21:31 ports/PortIndex_darwin_9_i386/PortIndex
%

Does anyone have any idea what’s going on here? It’s been the better part of a week now.

comment:9 Changed 11 years ago by jdgleeson

Cc: jdgleeson@… added

Cc Me!

comment:10 in reply to:  8 Changed 11 years ago by jmroot (Joshua Root)

Cc: wsiegrist@… added
Owner: changed from wsiegrist@… to admin@…

Replying to larryv@…:

Does anyone have any idea what’s going on here? It’s been the better part of a week now.

Presumably the job is not being run or is stuck for some reason. I don't know why Bill hasn't been able to respond.

comment:11 Changed 11 years ago by drkp (Dan Ports)

Cc: dports@… added

The distfiles mirror has also not been updating.

comment:12 Changed 11 years ago by mail@…

Cc: mail@… added

Cc Me!

comment:13 Changed 11 years ago by nonstop.server@…

Cc: nonstop.server@… added

Cc Me!

comment:14 Changed 11 years ago by wsiegrist@…

I'm investigating. Sorry for the delays, it is a combination of some vacation and hectic work loads.

comment:15 Changed 11 years ago by wsiegrist@…

Owner: changed from admin@… to cal@…

The sync script builds MacPorts first, and [107864] broke the build on Linux.

gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC rmd160cmd.c -o rmd160cmd.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC sha256cmd.c -o sha256cmd.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC readline.c -o readline.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC uid.c -o uid.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC tracelib.c -o tracelib.o
tracelib.c:49:23: error: sys/event.h: No such file or directory
tracelib.c: In function ‘send_file_map’:
tracelib.c:340: warning: implicit declaration of function ‘strlcat’
tracelib.c: In function ‘TracelibOpenSocketCmd’:
tracelib.c:517: error: ‘struct sockaddr_un’ has no member named ‘sun_len’
tracelib.c:519: error: ‘struct sockaddr_un’ has no member named ‘sun_len’
tracelib.c: At top level:
tracelib.c:541: error: array type has incomplete element type
tracelib.c: In function ‘TracelibRunCmd’:
tracelib.c:543: error: storage size of ‘kev’ isn’t known
tracelib.c:548: warning: implicit declaration of function ‘kqueue’
tracelib.c:572: warning: implicit declaration of function ‘EV_SET’
tracelib.c:572: error: ‘EVFILT_READ’ undeclared (first use in this function)
tracelib.c:572: error: (Each undeclared identifier is reported only once
tracelib.c:572: error: for each function it appears in.)
tracelib.c:572: error: ‘EV_ADD’ undeclared (first use in this function)
tracelib.c:572: error: ‘EV_RECEIPT’ undeclared (first use in this function)
tracelib.c:573: warning: implicit declaration of function ‘kevent’
tracelib.c:583: error: ‘EV_ERROR’ undeclared (first use in this function)
tracelib.c:692: error: ‘EV_EOF’ undeclared (first use in this function)
tracelib.c:543: warning: unused variable ‘kev’
make[2]: *** [tracelib.o] Error 1
make[2]: Leaving directory `/var/macports/trunk/base/src/pextlib1.0'
make[1]: *** [all] Error 1
make[1]: Leaving directory `/var/macports/trunk/base/src'
make: *** [all] Error 1

comment:16 Changed 11 years ago by wsiegrist@…

Actually, the time of the change and this bug don't support my theory. So maybe its not that change that started the problem, but that is where the sync script ends currently.

comment:17 Changed 11 years ago by jmroot (Joshua Root)

How is it after the latest changes to trunk? And can we add a Linux buildslave for base with the same environment as the server?

comment:18 Changed 11 years ago by skymoo (Adam Mercer)

Cc: ram@… added

Cc Me!

comment:19 Changed 11 years ago by jwhowse4

Cc: jwhowse4@… added

Cc Me!

comment:20 Changed 11 years ago by wsiegrist@…

Yes, we can get a linux slave going, I assume that would be base only?

Here is the latest error:

gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC strlcat.c -o strlcat.o
strlcat.c:23: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘strlcat’
make[2]: *** [strlcat.o] Error 1
make[2]: Leaving directory `/var/macports/trunk/base/src/pextlib1.0'
make[1]: *** [all] Error 1
make[1]: Leaving directory `/var/macports/trunk/base/src'
make: *** [all] Error 1

comment:21 in reply to:  20 ; Changed 11 years ago by larryv (Lawrence Velázquez)

Replying to wsiegrist@…:

Yes, we can get a linux slave going, I assume that would be base only?

It wouldn’t be a bad idea to have a ports buildslave also, for debugging issues with Portfile parsing and so forth.

comment:22 Changed 11 years ago by jmroot (Joshua Root)

Try after r108134.

comment:23 in reply to:  21 ; Changed 11 years ago by jmroot (Joshua Root)

Replying to larryv@…:

It wouldn’t be a bad idea to have a ports buildslave also, for debugging issues with Portfile parsing and so forth.

We don't want a ports buildslave like the others running on Linux, since a ton of stuff probably doesn't build. A new builder type that generates a PortIndex would be OK. Let's just go with base for now since that's really easy to set up.

comment:24 in reply to:  23 Changed 11 years ago by cooljeanius (Eric Gallager)

Replying to jmr@…:

Replying to larryv@…:

It wouldn’t be a bad idea to have a ports buildslave also, for debugging issues with Portfile parsing and so forth.

We don't want a ports buildslave like the others running on Linux, since a ton of stuff probably doesn't build.

How about one that just tries to build all the ports that include linux in their platform field in their portfiles?

Version 0, edited 11 years ago by cooljeanius (Eric Gallager) (next)

comment:25 Changed 11 years ago by stromnov (Andrey Stromnov)

Cc: stromnov@… added

Cc Me!

comment:26 Changed 11 years ago by jmroot (Joshua Root)

Bill, could you please incorporate r108145, r108146, and r108147 into the copy of mprsyncup on the server? That should fix the immediate problem. Please also delete the base checkouts for both trunk and release (should be /var/tmp/macports/trunk/base and /var/tmp/macports/release/base) and let it check out fresh copies, since there are some dylibs in there that we don't want on the rsync server.

comment:27 Changed 11 years ago by mojca (Mojca Miklavec)

Cc: mojca@… added

Cc Me!

comment:28 Changed 11 years ago by cjones051073 (Chris Jones)

Cc: jonesc@… added

Cc Me!

comment:29 Changed 11 years ago by wsiegrist@…

Next error:

gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC mktemp.c -o mktemp.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC realpath.c -o realpath.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC strlcat.c -o strlcat.o
gcc -shared -g -O2 -Wextra -Wall -pedantic      -fPIC -Wl,-soname,Pextlib.so Pextlib.o strsed.o fgetln.o md5cmd.o setmode.o xinstall.o fs-traverse.o strcasecmp.o vercomp.o filemap.o base32cmd.o sha1cmd.o curl.o rmd160cmd.o sha256cmd.o readline.o uid.o tracelib.o tty.o readdir.o pipe.o flock.o system.o mktemp.o realpath.o strlcat.o -o Pextlib.so -L/usr/lib64 -ltclstub8.5   -lcurl -lcrypto  ../registry2.0/registry.so
warning: error while loading Pextlib.so: couldn't load file "./Pextlib.so": registry.so: cannot open shared object file: No such file or directory
make[2]: *** [pkgIndex.tcl] Error 1
make[2]: Leaving directory `/var/macports/trunk/base/src/pextlib1.0'
make[1]: *** [all] Error 1
make[1]: Leaving directory `/var/macports/trunk/base/src'
make: *** [all] Error 1

comment:30 Changed 11 years ago by jmroot (Joshua Root)

We can fix that later; for now please just resolve as per comment:26.

comment:31 Changed 11 years ago by cooljeanius (Eric Gallager)

Someone in IRC said it was working again...

comment:32 Changed 11 years ago by wsiegrist@…

Owner: changed from cal@… to jmr@…

I applied the patches, looks like its working for me too. Josh, can you confirm?

comment:33 Changed 11 years ago by jmroot (Joshua Root)

Resolution: fixed
Status: newclosed

Yes, rsync and distfiles are updating again. Thanks.

comment:34 Changed 11 years ago by mndavidoff (Monte Davidoff)

Resolution: fixed
Status: closedreopened

It looks like it may have stopped working again. There have not been any updates for 16 hours.

$ TZ=UTC date
Wed Jul 17 07:01:35 UTC 2013
$ TZ=UTC rsync rsync://rsync.macports.org/release/tarballs/
drwxr-xr-x        4096 2012/07/20 20:37:09 .
-rw-r--r--     4915200 2013/07/16 14:57:37 base.tar
-rw-r--r--         512 2013/07/16 14:57:39 base.tar.rmd160
-rw-r--r--    53258240 2013/07/16 14:57:38 ports.tar
-rw-r--r--         512 2013/07/16 14:57:39 ports.tar.rmd160
drwxr-xr-x        4096 2013/07/16 14:57:20 PortIndex_darwin_10_i386
drwxr-xr-x        4096 2013/07/16 14:57:22 PortIndex_darwin_11_i386
drwxr-xr-x        4096 2013/07/16 14:57:25 PortIndex_darwin_12_i386
drwxr-xr-x        4096 2013/07/16 14:57:11 PortIndex_darwin_8_i386
drwxr-xr-x        4096 2013/07/16 14:57:08 PortIndex_darwin_8_powerpc
drwxr-xr-x        4096 2013/07/16 14:57:17 PortIndex_darwin_9_i386
drwxr-xr-x        4096 2013/07/16 14:57:14 PortIndex_darwin_9_powerpc

comment:35 Changed 11 years ago by wsiegrist@…

My fault, I left the mprsyncup job disabled after troubleshooting it manually. It is running now and should be done in about 30m.

comment:36 Changed 11 years ago by wsiegrist@…

Resolution: fixed
Status: reopenedclosed

We're caught up again and the sync job is enabled.

Note: See TracTickets for help on using tickets.