Opened 19 months ago

Closed 18 months ago

Last modified 13 months ago

#65809 closed defect (fixed)

py-nbconvert @7.0.0: install fails, due to build-time artifact fetch

Reported by: agl2015 Owned by: mascguy (Christopher Nielsen)
Priority: Normal Milestone:
Component: ports Version: 2.7.2
Keywords: Cc: artemiog@…, someuser12, reneeotten (Renee Otten), stromnov (Andrey Stromnov), cjones051073 (Chris Jones), ryandesign (Ryan Carsten Schmidt)
Port: py-nbconvert

Description (last modified by mascguy (Christopher Nielsen))

Build of py39-nbconvert on a 2014 27” iMac running the latest version of Catalina (10.15.7) fails when not able to retrieve css from:

https://unpkg.com/@jupyterlab/nbconvert-css@3.1.11/style/index.css.

I am attaching the main.log file for more info.

Attachments (1)

main.log.zip (4.4 KB) - added by agl2015 19 months ago.
main.log

Download all attachments as: .zip

Change History (17)

Changed 19 months ago by agl2015

Attachment: main.log.zip added

main.log

comment:1 Changed 19 months ago by jmroot (Joshua Root)

Keywords: py39-nbconvert removed
Owner: set to stromnov
Port: py-nbconvert added; py39-nbconvert removed
Status: newassigned

comment:2 Changed 19 months ago by mascguy (Christopher Nielsen)

Cc: mascguy added

comment:3 Changed 19 months ago by mascguy (Christopher Nielsen)

Cc: someuser12 added
Description: modified (diff)
Summary: py39-nbconvert fails to buildpy-nbconvert: build fails due to fetch SSL failures, occurring during build phase

This port was broken with update to 7.0.0, via this PR:

https://github.com/macports/macports-ports/pull/15943

Perhaps this can be fixed by explicitly fetching artifacts by the port itself, rather that letting it occur during the Python build...?

comment:4 Changed 19 months ago by mascguy (Christopher Nielsen)

Summary: py-nbconvert: build fails due to fetch SSL failures, occurring during build phasepy-nbconvert: build fails due to fetch SSL failures, occurring during Python build

comment:5 Changed 19 months ago by mascguy (Christopher Nielsen)

Cc: reneeotten added

Renee/All, is there any Python magic/helpers, to assist with projects that try to fetch via their build scripts? Or will we need to go through the usual port motions: declaring additional aliased master sites, distfiles, etc...?

comment:6 Changed 19 months ago by mascguy (Christopher Nielsen)

Summary: py-nbconvert: build fails due to fetch SSL failures, occurring during Python buildpy-nbconvert @7.0.0: install fails, due to build-time artifact fetch

The new approach in upstream version 7.x isn't ideal, as the CSS artifacts are now fetched at build time.

Last edited 19 months ago by mascguy (Christopher Nielsen) (previous) (diff)

comment:7 Changed 19 months ago by mascguy (Christopher Nielsen)

Reviewing what's done in other ports, I suppose we can handle this via direct use of wget/curl.

The following is semi-complete, apart from copying the artifacts to the appropriate location(s) in the work area. (It's slightly more involved than would otherwise be required, due to multiple artifacts named variables.css.)

    depends_fetch-append \
                    path:bin/wget:wget

    post-fetch {
        # Explicitly fetch CSS artifacts; all of this must remain in-sync with project build script
        set notebook_css_ver            5.4.0
        set jupyterlab_css_ver          3.1.11
        set jupyterlab_theme_light_ver  3.1.11
        set jupyterlab_theme_dark_ver   3.1.11

        set notebook_css_url \
            https://cdn.jupyter.org/notebook/${notebook_css_ver}/style/style.min.css
        set jupyterlab_css_url \
            https://unpkg.com/@jupyterlab/nbconvert-css@${jupyterlab_css_ver}/style/index.css
        set jupyterlab_theme_light_url \
            https://unpkg.com/@jupyterlab/theme-light-extension@${jupyterlab_theme_light_ver}/style/variables.css
        set jupyterlab_theme_dark_url \
            https://unpkg.com/@jupyterlab/theme-dark-extension@${jupyterlab_theme_dark_ver}/style/variables.css

        set css_fetch_dir \
            ${distpath}/css
        set wget_fetch_cmd \
            "${prefix}/bin/wget --no-verbose --no-check-certificate --timestamping"

        xinstall -d \
            ${css_fetch_dir}/notebook_css \
            ${css_fetch_dir}/jupyterlab_css \
            ${css_fetch_dir}/jupyterlab_theme_light \
            ${css_fetch_dir}/jupyterlab_theme_dark

        system -W ${css_fetch_dir}/notebook_css \
            "${wget_fetch_cmd} ${notebook_css_url}"
        system -W ${css_fetch_dir}/jupyterlab_css \
            "${wget_fetch_cmd} ${jupyterlab_css_url}"
        system -W ${css_fetch_dir}/jupyterlab_theme_light \
            "${wget_fetch_cmd} ${jupyterlab_theme_light_url}"
        system -W ${css_fetch_dir}/jupyterlab_theme_dark \
            "${wget_fetch_cmd} ${jupyterlab_theme_dark_url}"
    }

I'll leave this out here for a day or two, in case folks have any suggestions to improve/streamline the approach.

And ultimately we should reach out to upstream, to see if they can eliminate the need for all of this long-term.

comment:8 Changed 19 months ago by mascguy (Christopher Nielsen)

Cc: stromnov added; mascguy removed
Owner: changed from stromnov to mascguy

comment:9 Changed 19 months ago by mascguy (Christopher Nielsen)

Cc: cjones051073 added

Adding @cjones for awareness, since this is blocking builds of root6 and other dependent ports.

comment:10 Changed 19 months ago by cjones051073 (Chris Jones)

I personally really do not like the above...

Before restorting to anything like that, I think first we need to understand what is it about the buildbots causing problems, as I can build py310-nbconvert just fine locally

> sudo port -v build py310-nbconvert
<snip>
--->  Fetching distfiles for py310-nbconvert
--->  Verifying checksums for py310-nbconvert
--->  Checksumming nbconvert-7.0.0.tar.gz
--->  Extracting py310-nbconvert
--->  Extracting nbconvert-7.0.0.tar.gz
Executing:  cd "/opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work" && /usr/bin/gzip -dc '/opt/local/var/macports/distfiles/py-nbconvert/nbconvert-7.0.0.tar.gz' | /usr/bin/tar -xf - 
--->  Configuring py310-nbconvert
--->  Building py310-nbconvert
Executing:  cd "/opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0" && /opt/local/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 -m build --wheel --no-isolation --outdir /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work 
Downloading CSS: https://unpkg.com/@jupyterlab/nbconvert-css@3.1.11/style/index.css
Downloaded Notebook CSS to /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0/share/templates/lab/static/index.css
Downloading CSS: https://unpkg.com/@jupyterlab/theme-light-extension@3.1.11/style/variables.css
Downloaded Notebook CSS to /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0/share/templates/lab/static/theme-light.css
Downloading CSS: https://unpkg.com/@jupyterlab/theme-dark-extension@3.1.11/style/variables.css
Downloaded Notebook CSS to /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0/share/templates/lab/static/theme-dark.css
Downloading CSS: https://cdn.jupyter.org/notebook/5.4.0/style/style.min.css
Downloaded Notebook CSS to /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0/share/templates/classic/static/style.css
* Getting dependencies for wheel...
* Building wheel...
Successfully built nbconvert-7.0.0-py3-none-any.whl

so what is it about the buildbots causing the problem ?

comment:11 in reply to:  10 ; Changed 19 months ago by mascguy (Christopher Nielsen)

Replying to cjones051073:

Before resorting to anything like that, I think first we need to understand what is it about the buildbots causing problems

While that would be nice, the reality is that arbitrary artifacts shouldn't be fetched by the build script. Particularly not if we're interested in #ReproducibleBuilds.

Regardless of the SSL issue, we need to explicitly fetch and mirror these anyway. And I'm currently engaging with upstream, to see if this process can be simplified.

comment:12 in reply to:  11 Changed 19 months ago by mascguy (Christopher Nielsen)

Replying to mascguy:

While that would be nice, the reality is that arbitrary artifacts shouldn't be fetched by the build script. Particularly not if we're interested in #ReproducibleBuilds.

In case you or anyone else isn't aware of it, there have also been recent reports of bad actors introducing spyware and viruses in some Python components. And this is yet another reason to avoid arbitrary, uncontrolled downloads via build scripts.

comment:13 Changed 19 months ago by cjones051073 (Chris Jones)

I don't disagree, builds should not fetch during the build.

But still, why does this, as undesirable as it is, work outside the buildbots but not on them ?

comment:14 in reply to:  13 Changed 19 months ago by mascguy (Christopher Nielsen)

Cc: ryandesign added

Replying to cjones051073:

I don't disagree, builds should not fetch during the build.

But still, why does this, as undesirable as it is, work outside the buildbots but not on them ?

I'll let @ryandesign speak to that.

Last edited 19 months ago by mascguy (Christopher Nielsen) (previous) (diff)

comment:15 Changed 18 months ago by Christopher Nielsen <mascguy@…>

Resolution: fixed
Status: assignedclosed

In afe944bfb3b417fa12c6700814cffd6affb38189/macports-ports (master):

py-nbconvert: explicitly fetch css artifacts
Fixes: #65809

comment:16 Changed 13 months ago by ryandesign (Ryan Carsten Schmidt)

In 01f698f3b3b8e3ceeb4813b773af08518cf2246e/macports-ports (master):

py-nbconvert: Remove custom fetch code

Closes: #66011
See: #65809

Note: See TracTickets for help on using tickets.