the 'Available Ports' page is extremely out of date

[ray@…]

At the top of the page it says: "Index last updated:" and then nothing. So, when was the index last updated?

Also, I believe it would be useful to show information about the freshness of a port.

You could:

a) just show the date of the last update. b) show the update history, when each version of the port was updated.

Either would help. When I am looking at using a port, it makes a difference if one was updated a month ago and one was updated a year ago.

[toby@…]

* Bug 3270 has been marked as a duplicate of this bug. *

[blb@…]

Assigning to Juan since he says he's working on this.

[jmpp@…]

I'd like everyone to know that I am indeed working on this, but it's quite a complex issue, more than most think. To summarize, the script that generates the database (base/src/portmgr/ PortIndex2MySQL.tc) needs to be brought up to speed with the current darwinports sources for one (Kevin updated the API calls, but the script still doesn't detect non / based dp installations, i.e. a ~/ confined one) and then a special environment needs to be setup so that an automated job can handle keeping the db up to date nightly and submitting it to the web server (otherwise the job has to be handled manually, which is ubber lame!). The latter will require some changes to our OpenDarwin based services, namely a dedicated account that will handle this job and other automated chores like the index regen one. All this, and some other details which I'm skipping for brevety's sake, will take some time to complete, so please be patient. I will report here any progress though.

Thanks!

-jmpp

[yeled@…]

* Bug 5551 has been marked as a duplicate of this bug. *

[brennan@…]

(In reply to comment #3) Couldn't you just copy the text port files from the sync server and import them into a database? I don't think an actual dp installation (with /etc info, var/db etc) on the web server is necessary. Even some sort of text/hash database with heavy caching might suffice. In any event, appreciate your work on this.

[toby@…]

The DP installation doesn't need to be on the database or web server, it just needs to be somewhere. The script just generates a SQL command file which can be loaded into the MySQL database via any of the standard methods.

[pkern@…]

Can't you just sync the PortIndex out of the dports rsync tree and parse it with your script? Why do you need a DP installation? To use the tools DP ships, I think there's most probably DP anyway on the OD web server anyway just in case you really need them?

[yeled@…]

This is now fixed.

We have a script on sancho, that until today, was trying to use an *old* portindex to update the PortIndex from /usr/dports (which it cvs upped at 0230 each day)

the script now does this:

cd /usr/dports cvs -q up -PAd PortIndex2MySQL.tcl > /tmp/PortIndex.sql mysql -D darwinports -u foo --password=bar < /tmp/PortIndex.sql rm /tmp/PortIndex.sql cd ${darwinportsDocumentRoot}/downloads cvs -q up -PAd tar zcf darwinports-nightly-cvs-snapshot.tar.gz darwinports

I had to hand-fix the database for PortIndex2MySQL.tcl to work again, and now, if nothing happens, it should "just keep truckin'"

ATTN: can someone with a darwinports-www commit priv please apply the attached patch?

what this *doesn't* fix is the freshness of the Portfile. They are all still stale until I address that via portmgr@

[yeled@…]

please apply this patch in www/ for this bug to be *really* resolved.

[jmpp@…]

First off, I'd like to thank Charlie wholeheartedly for looking into this and actually taking care of it, we all appreciate it! (we all owe him at least a beer!)

A couple of questions though:

(In reply to comment #8)

This is now fixed.

We have a script on sancho, that until today, was trying to use an *old* portindex to update the PortIndex from /usr/dports (which it cvs upped at 0230 each day)

You mean an old dports/ tree or an old dp installation?

the script now does this:

  cd /usr/dports
  cvs -q up -PAd
  PortIndex2MySQL.tcl > /tmp/PortIndex.sql
  mysql -D darwinports -u foo --password=bar < /tmp/PortIndex.sql
  rm /tmp/PortIndex.sql
  cd ${darwinportsDocumentRoot}/downloads
  cvs -q up -PAd
  tar zcf darwinports-nightly-cvs-snapshot.tar.gz darwinports

In relation to my comment above, what are the two cvs updates (and therefore two cvs repos) for? The latter seems to be where we pull the nightly snapshot from; is the former the *old* portindex you refer to above? In any case, as I explained on IRC and in my original mail to you, the edits I made to the PortIndex2MySQL.tcl script should allow us to move this job away from the WebServer and only submit the final results. And for that matter we could also look into moving the nightly snapshot regen away from the webserver, taking care of it somewhere else and only uploading the final tarball.

All these jobs seem to entail their own cvs checkout and dports/ repo, but I'm sure we could concentrate them all on just one; I'm thinking about piggybagging all this on the setup Paul and I wrote to get the PortIndex regen job done (read the base/portmgr/IndexRegen.sh script). It should be doable...

I had to hand-fix the database for PortIndex2MySQL.tcl to work again, and now, if nothing happens,

it

should "just keep truckin'"

Is there a cron entry with the steps you detailed above? Can we expect the site to be kept up to date?

ATTN: can someone with a darwinports-www commit priv please apply the attached patch?

I will just now. Is this to fix the path to the Portfiles from the results page?

what this *doesn't* fix is the freshness of the Portfile. They are all still stale until I address that via portmgr@

I don't understand what you mean by this.... And since I'm portmgr@.. fire away! ;-) How come the Portfiles aren't fresh? Aren't they updated through the CVS commands above?

Well, again, thanks for the hard work! Regards,....

-jmpp

[yeled@…]

(In reply to comment #10)

First off, I'd like to thank Charlie wholeheartedly for looking into this and actually taking care of it, we all appreciate it! (we all owe him at least a beer!)

no wuckas.

A couple of questions though:

(In reply to comment #8)

This is now fixed.

We have a script on sancho, that until today, was trying to use an *old* portindex to update the PortIndex from /usr/dports (which it cvs upped at 0230 each day)

You mean an old dports/ tree or an old dp installation?

an old installation. (there are old/stale trees all over OD machines! it's very confusing.)

the script now does this:

  cd /usr/dports
  cvs -q up -PAd
  PortIndex2MySQL.tcl > /tmp/PortIndex.sql
  mysql -D darwinports -u foo --password=bar < /tmp/PortIndex.sql
  rm /tmp/PortIndex.sql
  cd ${darwinportsDocumentRoot}/downloads
  cvs -q up -PAd
  tar zcf darwinports-nightly-cvs-snapshot.tar.gz darwinports

In relation to my comment above, what are the two cvs updates (and therefore two cvs repos) for?

the /usr/dports tree is for (AFAICT) the "available ports" and maybe historically sancho's main tree.

the latter is where we pull the nightly snapshot from, and that's all i think.

to above? In any case, as I explained on IRC and in my original mail to you, the edits I made to the PortIndex2MySQL.tcl script should allow us to move this job away from the WebServer and only

yep this will be great. we will soon only use one tree. i promise. soon.

I had to hand-fix the database for PortIndex2MySQL.tcl to work again, and now, if nothing

happens,

it

should "just keep truckin'"

Is there a cron entry with the steps you detailed above? Can we expect the site to be kept up to date?

yes, the site will be kept up to date in 24 hour intervals.

ATTN: can someone with a darwinports-www commit priv please apply the attached patch?

I will just now. Is this to fix the path to the Portfiles from the results page?

yep.

what this *doesn't* fix is the freshness of the Portfile. They are all still stale until I address that via portmgr@

I don't understand what you mean by this.... And since I'm portmgr@.. fire away! ;-) How come the Portfiles aren't fresh? Aren't they updated through the CVS commands above?

the actual links to Portfiles is yet *another* CVS tree which hasn't been updated somehow.. again - once i move all the trees -> one this will be fixed too.

i wont email portmgr@ anymore. it is a trivial fix.

[vincent-opdarw@…]

The links in dependencies fail, at least those starting with "port:".

[yeled@…]

(In reply to comment #12)

The links in dependencies fail, at least those starting with "port:".

which one? works for me (inclined to close this bug)

[yeled@…]

oh i see. ill fix that.

[yeled@…]

I've fixed this, and also made the modification that when you click on a dependency it will show you the exact match, not a list of *matches*.

here is the diff (not the hackish regexp mod - unsure here)

Juan notes he likes the previous method of clicking a dependency and showing a list of matches for that string.

18:31 sancho:~% diff -u ports.php-default ports.php-charlie
--- ports.php-default   Sat Dec 10 18:31:00 2005
+++ ports.php-charlie   Sat Dec 10 18:30:43 2005
@@ -77,6 +77,9 @@
                if ($by == "name") {
                        $query = $query . " AND p.name LIKE '%" . addslashes($substr) . "%'";
                }
+               if ($by == "library") {
+                       $query = $query . " AND p.name='" . addslashes($substr) . "'";
+               }
                if ($by == "desc") {
                        $query = $query . " AND p.description LIKE '%" . addslashes($substr) . "%'";
                }
@@ -173,9 +176,9 @@
 <?
                                        while ( $nrow = mysql_fetch_array($nresult) ) {
                                                // lib:libpng.3:libpng -> libpng
-                                               $library = eregi_replace("^[^:]*:[^:]*:", "", $nrow[0]);
+               $library = eregi_replace("^([^:]*:[^:]*:|[^:]*:)", "", $nrow[0]);
                                        ?>
-                                               <a href="<?= $PHP_SELF; ?>?by=name&substr=<?= $library; ?>"><?= $library; ?></a>
+                                               <a href="<?= $PHP_SELF; ?>?by=library&substr=<?= $library; ?>"><?= $library; ?></a>
                                        <?
                                        }
                                }
zsh: 22288 exit 1     diff -u ports.php-default ports.php-charlie
18:33 sancho:~% 

[yeled@…]

(In reply to comment #15)

Juan notes he likes the previous method of clicking a dependency and showing a list of matches for that string.

i take that back. We're on the same page now,

[jmpp@…]

The site is back online and being udpated on a nightly basis off cron. The only pending items are strictly OpenDarwin organizational issues that do not directly affect the site's funcionality, so I'm closing this bug, finally!

Thanks again to Charlie for putting up with all my constant bugging and nagging!

-jmpp

[(none)]

Milestone Website & Documentation deleted