Improve mportsearch performance

[blb@…]

Attached is a patch which changes how mportsearch works:

• now it always reads in all of each PortIndex available in ${sources}
• goes through the cached data from that read to find possible matches

From a speedup point-of-view, it appears to increase the speed of multiple mportsearches in one run of port by 20%+. For example, running a dry-run install of gnome (port-test is a trunk-based installed with no ports installed):

$ time port-test -dy install gnome > gnome_orig.txt 2>&1

real    1m10.154s
user    1m0.102s
sys 0m9.089s

After the attached patch:

$ time port-test -dy install gnome > gnome_patch.txt 2>&1

real    0m50.689s
user    0m43.276s
sys 0m6.508s

The disadvantage is that memory usage increases; watching top for both of the above tests shows the original peaking at about 344M, the patched at about 419M (so about a 20% increase in memory usage).

The debug output from each is identical.

As the dependency tree has been increasing lately (use_* adding the proper deps, various X11-based changes) speeding this up is becoming more important.

[blb@…]

patch to base/src/macports1.0/macports.tcl

[jmr@…]

There's another approach that I'd like to compare against this. Stay tuned.

[jmr@…]

Here's a patch that adds a new "quick index" and a new mportlookup proc, which maps a single port name to its PortIndex entry.

[jmr@…]

Performance comparison:

Original
port outdated	52.61
sudo port -y install gnome	1:19.46
Quick index
port outdated	6.12
sudo port -y install gnome	19.31

[blb@…]

To compare against my earlier results, quickindex-array.diff shows:

$ time port-test -dy install gnome > gnome_patch2.txt 2>&1

real	0m26.863s
user	0m18.970s
sys	0m6.854s

using a max of 340M of memory. So faster than my update without the memory cost. Note that this was after the PortIndex.quick file had already been generated, though that definitely doesn't take much time either.

[blb@…]

Added new patch to change handling of quick_index variable and work better with multiple repositories.

[jmr@…]

Updated again: now being smart about not reading the quick index when it's just been generated, added some comments, renamed a couple of variables for clarity.

[blb@…]

New patch to fix issue with multiple repos

[blb@…]

Update my version again so it works with multiple repos, basically, quicklist in _mports_load_quickindex needs to be unset if it exists, otherwise the first repo's quicklist is used for the second repo.

[jmr@…]

Update: decompose generation of the quick index into its own proc, and call the new proc from the portindex script.

[jmr@…]

Update: change PortIndex.quick to have one entry per line, and update PortIndexRegen.sh to include PortIndex.quick.

[jmr@…]

Committed in r47110.