Base/ports tarballs and PortIndexes are not being updated

[mndavidoff (Monte Davidoff)]

It appears port sync is not downloading an up-to-date ports.tar. It seems to be stuck downloading a ports.tar that's currently about 20 hours old.

$ date
Sat Jul  6 10:45:29 PDT 2013
$ sudo port sync
--->  Updating the ports tree
$ ls -l /opt/local/var/macports/sources/rsync.macports.org/release/tarballs
total 130176
-rw-r--r--   1 root  admin   9759688 Jul  5 14:31 PortIndex
-rw-r--r--   1 root  admin       512 Jun 16 10:31 PortIndex.rmd160
drwxr-xr-x  24 root  wheel       816 Jan 31 09:30 base
-rw-r--r--   1 root  admin   3594240 Jul  5 15:03 base.tar
-rw-r--r--   1 root  admin       512 Jul  5 15:03 base.tar.rmd160
drwxr-xr-x  51 root  wheel      1734 Jul  6 10:46 ports
-rw-r--r--   1 root  admin  53278720 Jul  5 15:03 ports.tar
-rw-r--r--   1 root  admin       512 Jul  5 15:03 ports.tar.rmd160
$ date
Sat Jul  6 10:46:25 PDT 2013

[cooljeanius (Eric Gallager)]

cal might want to be cc-ed on this, as he brought up the same issue on macports-dev: ​https://lists.macosforge.org/pipermail/macports-dev/2013-July/023223.html

[neverpanic (Clemens Lang)]

I was actually just writing a mail for an issue originally reported in IRC. I think this is best assigned to our admins.

[su-v]

Cc Me!

[larryv (Lawrence Velázquez)]

It’s not just ports.tar.

% TZ=UTC rsync rsync://rsync.macports.org/release/tarballs/
drwxr-xr-x        4096 2012/07/20 20:37:09 .
-rw-r--r--     3594240 2013/07/05 22:03:03 base.tar
-rw-r--r--         512 2013/07/05 22:03:05 base.tar.rmd160
-rw-r--r--    53278720 2013/07/05 22:03:04 ports.tar
-rw-r--r--         512 2013/07/05 22:03:05 ports.tar.rmd160
drwxr-xr-x        4096 2013/07/05 22:02:16 PortIndex_darwin_10_i386
drwxr-xr-x        4096 2013/07/05 22:02:19 PortIndex_darwin_11_i386
drwxr-xr-x        4096 2013/07/05 22:02:21 PortIndex_darwin_12_i386
drwxr-xr-x        4096 2013/07/05 22:02:09 PortIndex_darwin_8_i386
drwxr-xr-x        4096 2013/07/05 22:02:06 PortIndex_darwin_8_powerpc
drwxr-xr-x        4096 2013/07/05 22:02:14 PortIndex_darwin_9_i386
drwxr-xr-x        4096 2013/07/05 22:02:11 PortIndex_darwin_9_powerpc
%

[b.forbort@…]

Cc Me!

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Cc Me!

[larryv (Lawrence Velázquez)]

Cc Me!

[larryv (Lawrence Velázquez)]

As ​John just noted, the daily tarball is out of date, too. Which makes sense, since it’s generated by the rsync server.

% curl -s https://distfiles.macports.org/ports.tar.gz | TZ=UTC pax -vzT 201307050000 | cut -c 43- | sort -r | head -n 20
Jul  5 22:02 ports/PortIndex_darwin_9_powerpc/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_9_powerpc
Jul  5 22:02 ports/PortIndex_darwin_9_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_9_i386
Jul  5 22:02 ports/PortIndex_darwin_8_powerpc/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_8_powerpc
Jul  5 22:02 ports/PortIndex_darwin_8_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_8_i386
Jul  5 22:02 ports/PortIndex_darwin_12_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_12_i386
Jul  5 22:02 ports/PortIndex_darwin_11_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_11_i386
Jul  5 22:02 ports/PortIndex_darwin_10_i386/PortIndex.quick
Jul  5 22:02 ports/PortIndex_darwin_10_i386
Jul  5 22:02 ports/PortIndex.quick
Jul  5 22:02 ports/PortIndex
Jul  5 21:31 ports/devel/qt4-creator-mac/Portfile
Jul  5 21:31 ports/devel/qt4-creator-mac
Jul  5 21:31 ports/PortIndex_darwin_9_powerpc/PortIndex
Jul  5 21:31 ports/PortIndex_darwin_9_i386/PortIndex
%

Does anyone have any idea what’s going on here? It’s been the better part of a week now.

[jdgleeson]

Cc Me!

[jmroot (Joshua Root)]

Replying to larryv@…:

Does anyone have any idea what’s going on here? It’s been the better part of a week now.

Presumably the job is not being run or is stuck for some reason. I don't know why Bill hasn't been able to respond.

[drkp (Dan Ports)]

The distfiles mirror has also not been updating.

[mail@…]

Cc Me!

[nonstop.server@…]

Cc Me!

[wsiegrist@…]

I'm investigating. Sorry for the delays, it is a combination of some vacation and hectic work loads.

[wsiegrist@…]

The sync script builds MacPorts first, and [107864] broke the build on Linux.

gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC rmd160cmd.c -o rmd160cmd.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC sha256cmd.c -o sha256cmd.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC readline.c -o readline.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC uid.c -o uid.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC tracelib.c -o tracelib.o
tracelib.c:49:23: error: sys/event.h: No such file or directory
tracelib.c: In function ‘send_file_map’:
tracelib.c:340: warning: implicit declaration of function ‘strlcat’
tracelib.c: In function ‘TracelibOpenSocketCmd’:
tracelib.c:517: error: ‘struct sockaddr_un’ has no member named ‘sun_len’
tracelib.c:519: error: ‘struct sockaddr_un’ has no member named ‘sun_len’
tracelib.c: At top level:
tracelib.c:541: error: array type has incomplete element type
tracelib.c: In function ‘TracelibRunCmd’:
tracelib.c:543: error: storage size of ‘kev’ isn’t known
tracelib.c:548: warning: implicit declaration of function ‘kqueue’
tracelib.c:572: warning: implicit declaration of function ‘EV_SET’
tracelib.c:572: error: ‘EVFILT_READ’ undeclared (first use in this function)
tracelib.c:572: error: (Each undeclared identifier is reported only once
tracelib.c:572: error: for each function it appears in.)
tracelib.c:572: error: ‘EV_ADD’ undeclared (first use in this function)
tracelib.c:572: error: ‘EV_RECEIPT’ undeclared (first use in this function)
tracelib.c:573: warning: implicit declaration of function ‘kevent’
tracelib.c:583: error: ‘EV_ERROR’ undeclared (first use in this function)
tracelib.c:692: error: ‘EV_EOF’ undeclared (first use in this function)
tracelib.c:543: warning: unused variable ‘kev’
make[2]: *** [tracelib.o] Error 1
make[2]: Leaving directory `/var/macports/trunk/base/src/pextlib1.0'
make[1]: *** [all] Error 1
make[1]: Leaving directory `/var/macports/trunk/base/src'
make: *** [all] Error 1

[wsiegrist@…]

Actually, the time of the change and this bug don't support my theory. So maybe its not that change that started the problem, but that is where the sync script ends currently.

[jmroot (Joshua Root)]

How is it after the latest changes to trunk? And can we add a Linux buildslave for base with the same environment as the server?

[skymoo (Adam Mercer)]

Cc Me!

[jwhowse4]

Cc Me!

[wsiegrist@…]

Yes, we can get a linux slave going, I assume that would be base only?

Here is the latest error:

gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC strlcat.c -o strlcat.o
strlcat.c:23: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘strlcat’
make[2]: *** [strlcat.o] Error 1
make[2]: Leaving directory `/var/macports/trunk/base/src/pextlib1.0'
make[1]: *** [all] Error 1
make[1]: Leaving directory `/var/macports/trunk/base/src'
make: *** [all] Error 1

[larryv (Lawrence Velázquez)]

Replying to wsiegrist@…:

Yes, we can get a linux slave going, I assume that would be base only?

It wouldn’t be a bad idea to have a ports buildslave also, for debugging issues with Portfile parsing and so forth.

[jmroot (Joshua Root)]

Try after r108134.

[jmroot (Joshua Root)]

Replying to larryv@…:

It wouldn’t be a bad idea to have a ports buildslave also, for debugging issues with Portfile parsing and so forth.

We don't want a ports buildslave like the others running on Linux, since a ton of stuff probably doesn't build. A new builder type that generates a PortIndex would be OK. Let's just go with base for now since that's really easy to set up.

[cooljeanius (Eric Gallager)]

Replying to jmr@…:

Replying to larryv@…:

It wouldn’t be a bad idea to have a ports buildslave also, for debugging issues with Portfile parsing and so forth.

We don't want a ports buildslave like the others running on Linux, since a ton of stuff probably doesn't build.

How about one that just tries to build all the ports that include linux in their platform field in their portfiles?

Edit: That happens to be the following ports:

Local-Admins-MacBook-Pro:RubyMotion ericgallager$ port echo platforms:linux
bodr                            
btrfs-progs                     
bzip2                           
cbmbasic                        
chemical-mime-data              
fizmo                           
gettext                         
gnupg12                         
go                              
gtk-doc                         
hexedit                         
libiconv                        
libmcs                          
libpng                          
logtalk                         
logtalk-devel                   
makepak                         
monit                           
mutt                            
netpbm                          
openbox                         
perl5                           
perl5.8                         
perl5.10                        
perl5.12                        
perl5.14                        
perl5.16                        
poppler                         
processing                      
py-pylibacl                     
py-xattr                        
py24-xattr                      
py25-pylibacl                   
py25-xattr                      
py26-pylibacl                   
py26-xattr                      
py27-pylibacl                   
py27-xattr                      
python24-doc                    
rox-all                         
rox-filer                       
rpm                             
rpm2cpio                        
rpm45                           
rpm50                           
rpm51                           
rpm52                           
rpm53                           
rpm54                           
tf                              
tom                             
xar-devel                       
xdg-utils                       
zeroinstall-injector            

That's just 54 ports so far, so that shouldn't be too many.

[stromnov (Andrey Stromnov)]

Cc Me!

[jmroot (Joshua Root)]

Bill, could you please incorporate r108145, r108146, and r108147 into the copy of mprsyncup on the server? That should fix the immediate problem. Please also delete the base checkouts for both trunk and release (should be /var/tmp/macports/trunk/base and /var/tmp/macports/release/base) and let it check out fresh copies, since there are some dylibs in there that we don't want on the rsync server.

[mojca (Mojca Miklavec)]

Cc Me!

[cjones051073 (Chris Jones)]

Cc Me!

[wsiegrist@…]

Next error:

gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC mktemp.c -o mktemp.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC realpath.c -o realpath.o
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic     -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fPIC strlcat.c -o strlcat.o
gcc -shared -g -O2 -Wextra -Wall -pedantic      -fPIC -Wl,-soname,Pextlib.so Pextlib.o strsed.o fgetln.o md5cmd.o setmode.o xinstall.o fs-traverse.o strcasecmp.o vercomp.o filemap.o base32cmd.o sha1cmd.o curl.o rmd160cmd.o sha256cmd.o readline.o uid.o tracelib.o tty.o readdir.o pipe.o flock.o system.o mktemp.o realpath.o strlcat.o -o Pextlib.so -L/usr/lib64 -ltclstub8.5   -lcurl -lcrypto  ../registry2.0/registry.so
warning: error while loading Pextlib.so: couldn't load file "./Pextlib.so": registry.so: cannot open shared object file: No such file or directory
make[2]: *** [pkgIndex.tcl] Error 1
make[2]: Leaving directory `/var/macports/trunk/base/src/pextlib1.0'
make[1]: *** [all] Error 1
make[1]: Leaving directory `/var/macports/trunk/base/src'
make: *** [all] Error 1

[jmroot (Joshua Root)]

We can fix that later; for now please just resolve as per comment:26.

[cooljeanius (Eric Gallager)]

Someone in IRC said it was working again...

[wsiegrist@…]

I applied the patches, looks like its working for me too. Josh, can you confirm?

[jmroot (Joshua Root)]

Yes, rsync and distfiles are updating again. Thanks.

[mndavidoff (Monte Davidoff)]

It looks like it may have stopped working again. There have not been any updates for 16 hours.

$ TZ=UTC date
Wed Jul 17 07:01:35 UTC 2013
$ TZ=UTC rsync rsync://rsync.macports.org/release/tarballs/
drwxr-xr-x        4096 2012/07/20 20:37:09 .
-rw-r--r--     4915200 2013/07/16 14:57:37 base.tar
-rw-r--r--         512 2013/07/16 14:57:39 base.tar.rmd160
-rw-r--r--    53258240 2013/07/16 14:57:38 ports.tar
-rw-r--r--         512 2013/07/16 14:57:39 ports.tar.rmd160
drwxr-xr-x        4096 2013/07/16 14:57:20 PortIndex_darwin_10_i386
drwxr-xr-x        4096 2013/07/16 14:57:22 PortIndex_darwin_11_i386
drwxr-xr-x        4096 2013/07/16 14:57:25 PortIndex_darwin_12_i386
drwxr-xr-x        4096 2013/07/16 14:57:11 PortIndex_darwin_8_i386
drwxr-xr-x        4096 2013/07/16 14:57:08 PortIndex_darwin_8_powerpc
drwxr-xr-x        4096 2013/07/16 14:57:17 PortIndex_darwin_9_i386
drwxr-xr-x        4096 2013/07/16 14:57:14 PortIndex_darwin_9_powerpc

[wsiegrist@…]

My fault, I left the mprsyncup job disabled after troubleshooting it manually. It is running now and should be done in about 30m.

[wsiegrist@…]

We're caught up again and the sync job is enabled.