Opened 19 months ago

Last modified 4 months ago

#55041 new enhancement

mpbb-list-subports is slow

Reported by: ryandesign (Ryan Schmidt) Owned by: admin@…
Priority: Normal Milestone:
Component: server/hosting Version:
Keywords: mpbb Cc: mojca (Mojca Miklavec)
Port:

Description (last modified by mojca (Mojca Miklavec))

Would be great if mpbb-list-subports could be made faster. Listing all subports of the php port takes 2 minutes 30 seconds on my 2016 MacBook Pro. Listing all of the py-* ports' subports took our buildbot 2 hours, listing the p5-* subports on the ppc buildbot took 26 hours. If we ever want to try to build all ports, at this rate, just listing the subports would probably take a week.

Change History (7)

comment:1 Changed 19 months ago by jmroot (Joshua Root)

This is a direct result of all the exclusion checks. Among other things, it queries the packages server with curl and runs port_binary_distributable.tcl for each subport.

comment:2 Changed 19 months ago by ryandesign (Ryan Schmidt)

The curl check is probably already as fast as it can get, since it's just checking the response code from an http request to a server on the same gigabit network.

What about the rest? Each tcl script we run, and we run several for each subport, has to start tclsh and initialize the ports system. Is there a way we can initialize the ports system just once per subport and get all the information we need?

We have $(sysctl -n hw.activecpu) CPU cores. Is there a way we can check that many ports in parallel?

Last edited 19 months ago by ryandesign (Ryan Schmidt) (previous) (diff)

comment:3 in reply to:  2 Changed 19 months ago by ryandesign (Ryan Schmidt)

Replying to ryandesign:

Is there a way we can check that many ports in parallel?

e.g. with GNU parallel?

comment:4 Changed 19 months ago by mojca (Mojca Miklavec)

Curl might be as fast as is gets, but it would be even faster if we had a local database with all available files.

I don't know enough about Tcl to be able to tell how to avoid re-initializing everything every single time.

comment:5 in reply to:  4 Changed 19 months ago by ryandesign (Ryan Schmidt)

Replying to mojca:

Curl might be as fast as is gets, but it would be even faster if we had a local database with all available files.

Maybe. Seems like something that could easily get out of sync. And how much time would this save? How quickly can we do a (SQLite?) database query? The curl check takes about 0.05 seconds, and it looks like we could cut that to 0.02 by using http instead of https.

I don't know enough about Tcl to be able to tell how to avoid re-initializing everything every single time.

I meant: instead of running archive-path.tcl and port_binary_distributable.tcl and portgroups.tcl and supported-archs.tcl for each subport, could we combine those into a single script? Move some of the logic from bash into Tcl? Even though we don't currently run all four scripts for all ports (some are run conditionally), I feel like probably most of the time is spent in initialization, such that combining them into a single script that does all four tasks would still be faster.

comment:6 Changed 12 months ago by mojca (Mojca Miklavec)

Cc: mojca added

comment:7 Changed 4 months ago by mojca (Mojca Miklavec)

Description: modified (diff)
Note: See TracTickets for help on using tickets.