Opened 15 years ago

Closed 15 years ago

#18259 closed enhancement (fixed)

Improve mportsearch performance

Reported by: blb@… Owned by: macports-tickets@…
Priority: Normal Milestone: MacPorts 1.8.0
Component: base Version: 1.8.0
Keywords: search mportsearch performance Cc: jmroot (Joshua Root)
Port:

Description

Attached is a patch which changes how mportsearch works:

  • now it always reads in all of each PortIndex available in ${sources}
  • goes through the cached data from that read to find possible matches

From a speedup point-of-view, it appears to increase the speed of multiple mportsearches in one run of port by 20%+. For example, running a dry-run install of gnome (port-test is a trunk-based installed with no ports installed):

$ time port-test -dy install gnome > gnome_orig.txt 2>&1

real    1m10.154s
user    1m0.102s
sys 0m9.089s

After the attached patch:

$ time port-test -dy install gnome > gnome_patch.txt 2>&1

real    0m50.689s
user    0m43.276s
sys 0m6.508s

The disadvantage is that memory usage increases; watching top for both of the above tests shows the original peaking at about 344M, the patched at about 419M (so about a 20% increase in memory usage).

The debug output from each is identical.

As the dependency tree has been increasing lately (use_* adding the proper deps, various X11-based changes) speeding this up is becoming more important.

Attachments (4)

macports.tcl.diff (7.9 KB) - added by blb@… 15 years ago.
patch to base/src/macports1.0/macports.tcl
quickindex.diff (11.8 KB) - added by jmroot (Joshua Root) 15 years ago.
quickindex-array-blb.diff (12.8 KB) - added by blb@… 15 years ago.
New patch to fix issue with multiple repos
quickindex-array.diff (14.1 KB) - added by jmroot (Joshua Root) 15 years ago.

Download all attachments as: .zip

Change History (14)

Changed 15 years ago by blb@…

Attachment: macports.tcl.diff added

patch to base/src/macports1.0/macports.tcl

comment:1 Changed 15 years ago by jmroot (Joshua Root)

There's another approach that I'd like to compare against this. Stay tuned.

Changed 15 years ago by jmroot (Joshua Root)

Attachment: quickindex.diff added

comment:2 Changed 15 years ago by jmroot (Joshua Root)

Cc: jmr@… added

Here's a patch that adds a new "quick index" and a new mportlookup proc, which maps a single port name to its PortIndex entry.

comment:3 Changed 15 years ago by jmroot (Joshua Root)

Performance comparison:

Original
port outdated52.61
sudo port -y install gnome1:19.46
Quick index
port outdated6.12
sudo port -y install gnome19.31

comment:4 Changed 15 years ago by blb@…

To compare against my earlier results, quickindex-array.diff shows:

$ time port-test -dy install gnome > gnome_patch2.txt 2>&1

real	0m26.863s
user	0m18.970s
sys	0m6.854s

using a max of 340M of memory. So faster than my update without the memory cost. Note that this was after the PortIndex.quick file had already been generated, though that definitely doesn't take much time either.

comment:5 Changed 15 years ago by blb@…

Added new patch to change handling of quick_index variable and work better with multiple repositories.

comment:6 Changed 15 years ago by jmroot (Joshua Root)

Updated again: now being smart about not reading the quick index when it's just been generated, added some comments, renamed a couple of variables for clarity.

Changed 15 years ago by blb@…

Attachment: quickindex-array-blb.diff added

New patch to fix issue with multiple repos

comment:7 Changed 15 years ago by blb@…

Update my version again so it works with multiple repos, basically, quicklist in _mports_load_quickindex needs to be unset if it exists, otherwise the first repo's quicklist is used for the second repo.

comment:8 Changed 15 years ago by jmroot (Joshua Root)

Update: decompose generation of the quick index into its own proc, and call the new proc from the portindex script.

Changed 15 years ago by jmroot (Joshua Root)

Attachment: quickindex-array.diff added

comment:9 Changed 15 years ago by jmroot (Joshua Root)

Update: change PortIndex.quick to have one entry per line, and update PortIndexRegen.sh to include PortIndex.quick.

comment:10 Changed 15 years ago by jmroot (Joshua Root)

Milestone: MacPorts FutureMacPorts 1.8.0
Resolution: fixed
Status: newclosed

Committed in r47110.

Note: See TracTickets for help on using tickets.