Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#62401 closed defect (fixed)

Import new builds into ports web site

Reported by: ryandesign (Ryan Carsten Schmidt) Owned by: neverpanic (Clemens Lang)
Priority: Normal Milestone:
Component: server/hosting Version:
Keywords: Cc: Dave-Allured (Dave Allured), cjones051073 (Chris Jones), FranklinYu (Franklin Yu), mascguy (Christopher Nielsen)
Port:

Description

The ports web site hasn't imported builds after Feb 20:

https://ports.macports.org/ports/all_builds/

That was the day the buildmaster's SSD died and it took me a week to find time to figure out how and where to bring the buildmaster back. But it's back now and has done many builds since then.

I wasn't able to recover data from the SSD so I had to go to backups which meant that information about some builds that happened (and that were already imported into the ports web site) was lost. I didn't know exactly how many builds were lost in each queue so I skipped ahead approximately 100 build numbers in each queue. That way anyone who clicks on a link to one of the lost builds will get a 404 error rather than information about an unrelated build (mostly: I let a couple builds go through before I skipped ahead).

If the web app keeps track of the last imported build number in each queue, maybe you could reset it to the last build number that we were able to restore:

Port Builder Build Number Start Time Elapsed Time Watcher Build Status
dyncall 10.6 48687 2021-02-20 14:12:11 0:05:57 15083 build successful
py39-u-msgpack-python 10.6_i386 26668 2021-02-20 13:08:50 0:05:59 15124 build successful
py38-mypy 10.7 44788 2021-02-20 13:07:41 0:01:57 15121 build successful
py38-mypy 10.8 38711 2021-02-20 14:11:21 0:02:00 12979 build successful
gcc-devel 10.9 138837 2021-02-20 13:49:32 1:47:07 42385 build successful
go 10.10 135541 2021-02-20 11:17:51 0:01:54 42712 failed install-port
postgresql95-server 10.11 134031 2021-02-20 11:48:14 0:01:43 42651 build successful
fluent-bit 10.12 139112 2021-02-20 13:02:58 0:04:50 41647 build successful
go 10.13 103763 2021-02-20 12:59:20 0:10:26 33613 build successful
go 10.14 79558 2021-02-20 13:12:31 0:10:40 24308 build successful
squirrel-ime 11_arm64 12483 2021-02-20 14:55:18 0:01:59 3270 failed install-port
octave-nan 11_x86_64 22001 2021-02-20 15:50:11 0:16:04 3097 build successful

And then import starting after there. If the current import process checks successive build numbers and stops when it gets a 404 error, please do a one-time change that continues on through the 404 errors for up to 100 or so further build numbers in each queue to find where we started again after the skip, and then import normally from there.

10.15 is not in the table because 10.15 hasn't done any builds since then and I didn't skip its build numbers ahead. The 10.15 builder is offline for unrelated reasons and will come back eventually.

The buildmaster is temporarily running on a very slow disk so if your import process normally performs multiple requests to the server simultaneously, please limit it to just one request at a time for now, and it would also help to delay 10 seconds between each request to give the disk room to serve other requests that might be coming in from GitHub to schedule builds.

Change History (11)

comment:1 Changed 3 years ago by neverpanic (Clemens Lang)

Owner: changed from admin@… to neverpanic
Status: newaccepted

I believe this is happening because the webapp has a bug when a port only changes in case, not in name, which causes an SQL exception in the transaction and stops importing new ports.

I've been getting emails about this for a while now, and I have fixed a similar case manually before, but it was such a long time ago that I don't remember how I did it and will have to re-read to code to find out.

Unfortunately, I haven't gotten around to doing that.

comment:2 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Ah ok. Yes some case-only port renames have been happening lately and while I'm in favor of the web app handling port names case-insensitively it does need to be able to handle case-only renames.

comment:3 Changed 3 years ago by arjunsalyan (Arjun Salyan)

The duplicate ports can be identified by looking for the following line(s) in the logs of the update cronjob:

Could not append {1} as a dependency to {2}, because the depspec does not uniquely identify a port.

And then we need to delete the duplicate entries from the ports table.

The presence of the new version (which fixes all of these issues) prevents me from making changes to the existing version. But this one keeps repeating and might better be fixed.

comment:4 Changed 3 years ago by Dave-Allured (Dave Allured)

Cc: Dave-Allured added

comment:5 Changed 3 years ago by cjones051073 (Chris Jones)

Cc: cjones051073 added

comment:6 Changed 3 years ago by FranklinYu (Franklin Yu)

Cc: FranklinYu added

comment:7 in reply to:  description Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to ryandesign:

The buildmaster is temporarily running on a very slow disk so if your import process normally performs multiple requests to the server simultaneously, please limit it to just one request at a time for now, and it would also help to delay 10 seconds between each request to give the disk room to serve other requests that might be coming in from GitHub to schedule builds.

Buildmaster is back on a fast SSD now so feel free to access it as much and as frequently as needed.

comment:9 Changed 3 years ago by neverpanic (Clemens Lang)

In addition to the two known cases, there are also a few other case-only renames that now cause the same problem again:

webapp=> select count(*), lower(name) from ports_port group by lower(name) having count(*) > 1;
 count |      lower
-------+-----------------
     2 | transformerkit
     2 | hnhui
     2 | py35-nuitka
     2 | py34-whoosh
     2 | py36-nuitka
     2 | keepasskit
     2 | kissxml
     2 | py34-sqlalchemy
     2 | py27-nuitka
(9 rows)

comment:10 Changed 3 years ago by neverpanic (Clemens Lang)

Resolution: fixed
Status: acceptedclosed

comment:11 Changed 3 years ago by mascguy (Christopher Nielsen)

Cc: mascguy added
Note: See TracTickets for help on using tickets.