Opened 12 years ago

Closed 12 years ago

#33707 closed defect (fixed)

py-novas_py fetches files during its build phase

Reported by: ryandesign (Ryan Carsten Schmidt) Owned by: lpsinger (Leo Singer)
Priority: Normal Milestone:
Component: ports Version: 2.0.4
Keywords: Cc:
Port: py-novas_py

Description

py-novas_py fetches files during its build phase, which is troublesome and should be avoided if possible. (Fetch during the fetch phase instead.)

For some systems with more restrictive networks, like the buildbot, this even causes a build failure:

http://build.macports.org/builders/buildports-snowleopard-x86_64/builds/5749/steps/compile/logs/stdio

Attachments (3)

Portfile (12.1 KB) - added by lpsinger (Leo Singer) 12 years ago.
new Portfile
patch-setup.py.diff (455 bytes) - added by lpsinger (Leo Singer) 12 years ago.
additional patch file
Portfile.diff (9.9 KB) - added by ryandesign (Ryan Carsten Schmidt) 12 years ago.

Download all attachments as: .zip

Change History (17)

comment:1 Changed 12 years ago by lpsinger (Leo Singer)

NOVAS_Py's setup.py script downloads the entire contents of the directory ftp://ssd.jpl.nasa.gov/pub/eph/planets/ascii/de405. Is there a way to retrieve all of the contents of a directory during the fetch phase, or do I need to override or augment the fetch phase?

As an alternative, I could hard-code the paths of each of the files; the last modification times in FTP are all in 2007.

comment:2 Changed 12 years ago by ryandesign (Ryan Carsten Schmidt)

Each file that needs to be downloaded would be listed in the port's distfiles line, and checksums for each file would be listed as well. Then if necessary in post-extract you would move these files into places where the build script will recognize them, perhaps even needing to patch the build script to not download the files.

If the distfile names do not contain the novas_py version number (and these don't), then you have a problem with unversioned distfiles and need to set dist_subdir, but that would then cause unnecessary re-downloading and re-mirroring of all those files when the novas_py version is increased but those other files don't change.

How are these files used by novas_py? Are they merely installed into a particular location and used at runtime? If so, perhaps a separate port could be created that only installs these data files, and novas_py could then depend on it. That new port (novas_py-data?) could set dist_subdir, but wouldn't need to be updated until those files actually change.

comment:3 in reply to:  2 Changed 12 years ago by lpsinger (Leo Singer)

Replying to ryandesign@…:

How are these files used by novas_py?

These are ASCII data files generated by JPL (they are a solar system ephemeris, or a table of the calculated positions of solar system objects, spanning many centuries). The NOVAS_Py setup.py script downloads them and packages them into a single binary file that is used by the underlying C NOVAS library. There are certain NOVAS calls that don't require an ephemeris, but many do. Most users would want to have an ephemeris.

To complicate matters, the code for packing the ASCII files into the one binary file lives in the NOVAS_Py package.

Another option would be for a third party (e.g., myself) to generate a binary ephemeris file, with a filename that reflects the date on which the JPL data files were downloaded. Then a Port could be created for this. Yet another complication: the file's endianness matters; it has to match the native endianness of the platform.

comment:4 Changed 12 years ago by ryandesign (Ryan Carsten Schmidt)

Simplest then is to just list all the files as distfiles, list their checksums, and inform the build script to use those files (possibly patching it). If the data files ever change, then we can add "dist_subdir ${name}/${version}" but since the files haven't changed in years we can skip that for now.

comment:5 in reply to:  4 Changed 12 years ago by lpsinger (Leo Singer)

Replying to ryandesign@…:

Simplest then is to just list all the files as distfiles, list their checksums, and inform the build script to use those files (possibly patching it). If the data files ever change, then we can add "dist_subdir ${name}/${version}" but since the files haven't changed in years we can skip that for now.

If I don't specify dist_subdir, will those files just be silently carried over from one version of the Port to another? That is exactly the behavior that I want.

comment:6 Changed 12 years ago by ryandesign (Ryan Carsten Schmidt)

Yes, and that won't be a problem, unless upsteam at some point updates one of those files.

Changed 12 years ago by lpsinger (Leo Singer)

Attachment: Portfile added

new Portfile

Changed 12 years ago by lpsinger (Leo Singer)

Attachment: patch-setup.py.diff added

additional patch file

comment:7 Changed 12 years ago by lpsinger (Leo Singer)

I just added a new Portfile and one extra patch file implementing what was suggested.

comment:8 Changed 12 years ago by ryandesign (Ryan Carsten Schmidt)

Looks good to me. I'm attaching a diff of your Portfile changes, with the following changes:

  • simplify post-extract using eval ... glob
  • use "-W" to change directory in a "system" command instead of manually calling "cd"

Changed 12 years ago by ryandesign (Ryan Carsten Schmidt)

Attachment: Portfile.diff added

comment:9 Changed 12 years ago by lpsinger (Leo Singer)

Resolution: fixed
Status: newclosed

Fixed in r91231. Thanks for your help!

comment:10 Changed 12 years ago by lpsinger (Leo Singer)

Resolution: fixed
Status: closedreopened

The buildbot fails when attempting to retrieve the JPL ephemeris files via FTP. Is this a problem with this Port?

comment:11 Changed 12 years ago by mf2k (Frank Schima)

Unfortunately the buildbot cannot download from FTP sources. This is a general issue and has to do with security. Ideally a http source can be added for this port.

comment:12 in reply to:  11 Changed 12 years ago by lpsinger (Leo Singer)

Replying to macsforever2000@…:

Unfortunately the buildbot cannot download from FTP sources. This is a general issue and has to do with security. Ideally a http source can be added for this port.

I just checked; I do not think that there is an alternate http URL.

comment:13 Changed 12 years ago by ryandesign (Ryan Carsten Schmidt)

The buildbot was fixed to be able to download from FTP sources some time ago; I believe Daniel Luke provided a proxy server. So I don't know why this is now failing.

comment:14 Changed 12 years ago by jmroot (Joshua Root)

Resolution: fixed
Status: reopenedclosed
Note: See TracTickets for help on using tickets.