Opened 23 months ago

Last modified 8 weeks ago

#48807 new defect

python messes with terminal state on exit

Reported by: yselivanov@… Owned by: jyrkiwahlstedt
Priority: Normal Milestone:
Component: ports Version: 2.3.3
Keywords: haspatch Cc: MarcusCalhoun-Lopez, su-v, Themanwithoutaplan, eborisch (Eric A. Borisch), jambonrose (Andrew Pinkham), jyrkiwahlstedt, smithsp (Sterling Smith), kurtjaeke@…, steve+macports@…, dualityim@…, cdeil (Christoph Deil), nathanfriday@…, mleinartas@…, fhgwright (Fred Wright), bpanulla (Brian Panulla), xfacter@…, brett@…, Russell-Jones-OxPhys (Russell Jones), sporring@…, mojca (Mojca Miklavec), ryandesign (Ryan Schmidt), zaxdo@…, larryv (Lawrence Velázquez), icemac (Michael Howitz), suominen (Kimmo Suominen), rogererens (Roger Erens), lesinigo (Luca Lesinigo), dsedivec, aikchar (Hamza Sheikh), mohd-akram (Mohamed Akram), reubano (Reuben Cummings), davidchambers (David Chambers), 1-61803, aminggs (Andrew Inggs), michaellass (Michael Lass), stromnov (Andrew Stromnov), ibrahimkarahan, jamadden (Jason Madden)
Port: python26 python27 python python34 python35 libedit

Description

Here's what happens after you exit the repl:

yury@ysmac ~ $ python3.5
Python 3.5.0rc3 (default, Sep  7 2015, 23:03:25)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D>>>
yury@ysmac ~ $ yury@ysmac ~ $ yury@ysmac ~ $ yury@ysmac ~ $ yury@ysmac ~ $

Attachments (5)

readline.patch (2.1 KB) - added by eborisch (Eric A. Borisch) 23 months ago.
Patches python 26,27,34,35 to have default +readline variant
python-notes.patch (2.5 KB) - added by eborisch (Eric A. Borisch) 15 months ago.
py27.patch (1.8 KB) - added by eborisch (Eric A. Borisch) 13 months ago.
Patch against r148368; reverts to libedit but leaves notes (install py27-readline suggestion) intact.
readline_c.diff (288 bytes) - added by eborisch (Eric A. Borisch) 13 months ago.
Reverse of upstream 1.111 changes.
patch-libedit.diff (7.3 KB) - added by mohd-akram (Mohamed Akram) 5 months ago.
Updated libedit patch

Download all attachments as: .zip

Change History (132)

comment:1 Changed 23 months ago by yselivanov@…

You also can't see what you type.

Typing 'reset[ENTER]' fixes the terminal. It works this way both in iTerm and in stock Terminal.app

Here's some output from python3.4:

yury@ysmac ~ $ python3
Python 3.4.3 (default, Aug 19 2015, 17:29:49)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
yury@ysmac ~ $
yury@ysmac ~ $
yury@ysmac ~ $
yury@ysmac ~ $
yury@ysmac ~ $

comment:2 Changed 23 months ago by yselivanov@…

  • Cc yselivanov@… added

Cc Me!

comment:3 Changed 23 months ago by mf2k (Frank Schima)

  • Cc yselivanov@… removed
  • Keywords python removed
  • Owner changed from macports-tickets@… to jwa@…

In the future, please Cc the port maintainers (port info --maintainers python35), if any. As reporter, you do not need to Cc yourself.

comment:4 Changed 23 months ago by su-v

On OS X 10.7.5, all python versions rebuilt with ncurses 6.0 are affected, see also:
https://lists.macosforge.org/pipermail/macports-dev/2015-August/031308.html

comment:5 Changed 23 months ago by su-v

  • Cc suv-sf@… added

Cc Me!

comment:6 Changed 23 months ago by Themanwithoutaplan

  • Cc charlie.clark@… added

Cc Me!

comment:7 Changed 23 months ago by Themanwithoutaplan

I suspect it's the related to the same issue but within the Python shell I need to press Return a second time to get a command prompt. This is a serious inconvenience!

comment:8 follow-up: Changed 23 months ago by yselivanov@…

Final Python 3.5 will be released in a few days. Is there a chance this will be fixed?

comment:9 Changed 23 months ago by musicant@…

  • Cc musicant@… added

Cc Me!

comment:10 in reply to: ↑ 8 ; follow-up: Changed 23 months ago by Themanwithoutaplan

Replying to yselivanov@…:

Final Python 3.5 will be released in a few days. Is there a chance this will be fixed?

I don't think that really matters. This is affecting all versions of Python.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 23 months ago by ufdup89@…

reverting to ncurses @5.9_2 seems to fix the problem...

Replying to charlie.clark@…:

Replying to yselivanov@…:

Final Python 3.5 will be released in a few days. Is there a chance this will be fixed?

I don't think that really matters. This is affecting all versions of Python.

Last edited 23 months ago by ufdup89@… (previous) (diff)

comment:12 in reply to: ↑ 11 Changed 23 months ago by Themanwithoutaplan

Replying to ufdup89@…:

reverting to ncurses @5.9_2 seems to fix the problem...

Sure, but this isn't easily done because you can't downgrade ports. But it does suggest that there may be a bug in ncurses v6.

comment:13 Changed 23 months ago by eborisch (Eric A. Borisch)

As noted in the mailing on the list, adjusting the port to use readline rather than libedit makes things work.

comment:14 Changed 23 months ago by eborisch (Eric A. Borisch)

  • Cc eborisch@… added

Cc Me!

comment:15 Changed 23 months ago by eborisch (Eric A. Borisch)

  • Summary changed from python35 messes with terminal state on exit to python messes with terminal state on exit

comment:16 Changed 23 months ago by jambonrose (Andrew Pinkham)

  • Cc code@… added

Cc Me!

comment:17 Changed 23 months ago by eborisch (Eric A. Borisch)

Just copying in my original email about this (with a workaround) :

I've recently noticed (not sure when it changed) that when I enter and then exit() the python (using python27 in particular) interpreter built against libedit, the tty flags (as reported by stty -a) aren't getting reset when exiting python -- most noticeably the echo flag is getting turned off. Yes, yes, reset will fix it, but still.

If I build python27 against libreadline instead of libedit (by disabling the libedit patch file) it works as expected.

comment:18 Changed 23 months ago by eborisch (Eric A. Borisch)

There is a stackoverflow thread as well: http://stackoverflow.com/questions/32578009/python-macports-and-buffer-problems - I've posted a link to this discussion there; completing the loop.

comment:19 follow-ups: Changed 23 months ago by eborisch (Eric A. Borisch)

Does it work for those impacted if you install pyNN-readline?

comment:20 in reply to: ↑ 19 Changed 23 months ago by Themanwithoutaplan

Replying to eborisch@…:

Does it work for those impacted if you install pyNN-readline?

I confirm that this works for Python 2.6, Python 2.7 and Python 3.4. But it doesn't work for Python 3.3 (port is obsolete) or Python 3.5 (no port py35-readline).

comment:21 in reply to: ↑ 19 Changed 23 months ago by su-v

Replying to eborisch@…:

Does it work for those impacted if you install pyNN-readline?

On OS X 10.7.5:

  • Confirmed for Python 2.6, 2.7 and 3.4
  • Not confirmed for Python 2.5 (I reinstated py25-readline in a local portfile repository for testing)

comment:22 Changed 23 months ago by jmroot (Joshua Root)

  • Cc mcalhoun@… added
  • Port libedit added

comment:23 Changed 23 months ago by eborisch (Eric A. Borisch)

As mentioned on the mailing list, this addresses the symptom, and has not identified the underlying issue. For users who just want it to work, that is likely OK, but we should still try to understand the root cause.

comment:24 follow-up: Changed 23 months ago by yselivanov@…

It's nearly impossible to use python right now. Can this be fixed asap? If switching to readline from libedit helps -- can we do that?

comment:25 Changed 23 months ago by eborisch (Eric A. Borisch)

Users can install pyNN-readline in the interim. The problem doesn't exist for everyone (see mailing list) so it's not clear what the best course of action is here...

comment:26 in reply to: ↑ 24 Changed 23 months ago by Themanwithoutaplan

Replying to yselivanov@…:

It's nearly impossible to use python right now. Can this be fixed asap? If switching to readline from libedit helps -- can we do that?

Python 3.4 is usable once you install py34-readline. Unfortunately, there is no py35-readline but I'm not sure what you'd be using Python 3.5 for at the moment anyway (other than compatibility testing). If you really need it today then download the one from python.org until this bug is fixed.

Could you please add Python 2.6, 2.7, Python 3.3 and Python 3.4 to the ports affected by the bug?

comment:27 Changed 23 months ago by eborisch (Eric A. Borisch)

  • Cc jwa@… added
  • Port python26 python27 python python34 added

comment:28 follow-up: Changed 23 months ago by eborisch (Eric A. Borisch)

I've added py35-readline, so if you 'port sync' you should be able to install that now.

I would propose adding the following variant (and likely making it default) on all python flavors until this is resolved.

+variant readline description {Use readline instead of libedit} {
+    patchfiles-delete       patch-libedit.diff
+    depends_lib-append      port:readline
+    depends_lib-delete      port:libedit
+}
+

comment:29 in reply to: ↑ 28 ; follow-up: Changed 23 months ago by Themanwithoutaplan

Replying to eborisch@…:

I've added py35-readline, so if you 'port sync' you should be able to install that now.

Thanks very much but not there yet. Could you add py33-readline for completeness? I tend to work in 3.4 or 2.7 and only use the other versions in tox.

Wouldn't another solution be to go back to the older version of ncurses? Or is that likely to cause conflicts with other ports that depend on the newer one?

comment:30 in reply to: ↑ 29 Changed 23 months ago by eborisch (Eric A. Borisch)

Replying to charlie.clark@…:

Replying to eborisch@…:

I've added py35-readline, so if you 'port sync' you should be able to install that now.

Thanks very much but not there yet. Could you add py33-readline for completeness? I tend to work in 3.4 or 2.7 and only use the other versions in tox.

I'd prefer not; it's already in the python graveyard (use 3.4 instead).

Wouldn't another solution be to go back to the older version of ncurses? Or is that likely to cause conflicts with other ports that depend on the newer one?

I don't think that's the best course. I'd much rather see the readline variant above until the libedit/ncurses issue is resolved -- or identified, really. Still unclear as to why not everyone is having issues.

comment:31 follow-up: Changed 23 months ago by ryandesign (Ryan Schmidt)

#48911 may be related.

comment:32 in reply to: ↑ 31 Changed 23 months ago by Themanwithoutaplan

Replying to ryandesign@…:

#48911 may be related.

Yep, that's the same issue.

comment:33 Changed 23 months ago by eborisch (Eric A. Borisch)

Given there have been no insights or suggestions on how to fix python w/libedit issues, I've attached a proposed patch that links python against readline -- it is done as a default variant +readline, so it should be easy to un-do later once the underlying issues are identified and resolved.

Any objections to committing this?

Changed 23 months ago by eborisch (Eric A. Borisch)

Patches python 26,27,34,35 to have default +readline variant

comment:34 Changed 23 months ago by eborisch (Eric A. Borisch)

I added +readline (not on by default) in r140531. Please 'port selfupdate' and then 'port install python(26|27|34|35) +readline' if you are experiencing these issues.

comment:35 Changed 22 months ago by icemac (Michael Howitz)

  • Cc mh@… added

Cc Me!

comment:36 Changed 22 months ago by dh@…

This issue also affects the debugger (pdb), which often fails to print its prompt after a command.

comment:37 follow-up: Changed 22 months ago by eborisch (Eric A. Borisch)

Fixed when Python is installed +readline?

comment:38 in reply to: ↑ 37 Changed 22 months ago by icemac (Michael Howitz)

Replying to eborisch@…:

Fixed when Python is installed +readline?

Yes, +readline helped for me for the python console and the debugger.

comment:39 Changed 22 months ago by smithsp (Sterling Smith)

  • Cc smithsp@… added

Cc Me!

comment:40 Changed 22 months ago by kurtjaeke@…

  • Cc kurtjaeke@… added

Cc Me!

comment:41 Changed 21 months ago by ryandesign (Ryan Schmidt)

Has duplicates #48911, #49479.

comment:42 follow-up: Changed 21 months ago by eborisch (Eric A. Borisch)

  • Keywords haspatch added

As I noted above, by not patching in libedit, this problem goes away. This is available via the +readline variant since r140531 (September 21, 2015).

Is there some objection to making +readline a default_variant, or perhaps going one step forward and making +libedit the variant, and make using libreadline the stock? (This is important wrt. what people get when using 'port upgrade outdated'.)

This is clearly continuing to bite people. Is there (potentially? haven't investigated) some licensing benefit to libedit vs. libreadline?

Last edited 21 months ago by eborisch (Eric A. Borisch) (previous) (diff)

comment:43 in reply to: ↑ 42 Changed 21 months ago by Themanwithoutaplan

Replying to eborisch@…:

Is there (potentially? haven't investigated) some licensing benefit to libedit vs. libreadline?

Yes, readline is GPL3 which a serious encumbrance. So we really need to fix what went wrong in libedit.

comment:44 follow-up: Changed 21 months ago by eborisch (Eric A. Borisch)

Really?

When I run a distributable check (/opt/mports/portmgr/jobs/port_binary_distributable.tcl -v <port> <variants>) on the ports I have installed that depend on python27 (I have over 200 installed that match; admittedly a subset) the only ones that lose distributability are gd2, graphviz[-gui], and quartz-wm. Having to build those from source doesn't seem (to me) to be too high a cost to pay for working python.

comment:45 in reply to: ↑ 44 Changed 21 months ago by Themanwithoutaplan

Replying to eborisch@…:

When I run a distributable check (/opt/mports/portmgr/jobs/port_binary_distributable.tcl -v <port> <variants>) on the ports I have installed that depend on python27 (I have over 200 installed that match; admittedly a subset) the only ones that lose distributability are gd2, graphviz[-gui], and quartz-wm. Having to build those from source doesn't seem (to me) to be too high a cost to pay for working python.

Multiply by the number of Python versions you have installed (5 on my system…) And who has the time and energy to double-check this all the time?

Anyway, GPL3 isn't just about redistribution and linking but let's not go down that particular rabbit hole.

What I don't understand is why nothing else seems to have been so affected by the changes in libedit. But that is where we need to focus our efforts.

comment:46 Changed 21 months ago by steve+macports@…

  • Cc steve+macports@… added

Cc Me!

comment:47 Changed 21 months ago by dualityim@…

  • Cc dualityim@… added

Cc Me!

comment:48 Changed 21 months ago by cdeil (Christoph Deil)

  • Cc Deil.Christoph@… added

Cc Me!

comment:49 Changed 21 months ago by ryandesign (Ryan Schmidt)

  • Cc nathanfriday@… added

Has duplicate #49739.

comment:50 Changed 21 months ago by eborisch (Eric A. Borisch)

Ok. Can we put a note in the python install that "if you experience issues with your terminal after exiting Python, please install pyNN-readline or 'pythonNN +readline' and report your issues on #48807." (With the actual URL.)

Or can we (for now) change to readline and make +libedit a variant. If the licensing is such an issue, perhaps this will motivate someone to dig into the underlying issue. Once the issue is resolved, switch back. It seems silly to put out a broken version because we like the licensing better. Any technical objections to doing this?

comment:51 Changed 20 months ago by mleinartas@…

  • Cc mleinartas@… added

Cc Me!

comment:52 Changed 19 months ago by seanfarley (Sean Farley)

Just had another report about this on the mercurial channel. What's the status with changing to readline and making +libedit a variant?

comment:53 Changed 19 months ago by seanfarley (Sean Farley)

Anyone objected to me making the readline change?

comment:54 Changed 19 months ago by fhgwright (Fred Wright)

  • Cc fw@… added

Cc Me!

comment:55 Changed 19 months ago by bpanulla (Brian Panulla)

  • Cc macports@… added

Cc Me!

comment:56 Changed 18 months ago by xfacter@…

  • Cc xfacter@… added

Cc Me!

comment:57 follow-up: Changed 18 months ago by piannucci (Peter Iannucci)

Or we can just roll back. I just built and installed libedit from the portfile at rev. 127130, and it's working correctly with Python 3.4.

comment:58 in reply to: ↑ 57 Changed 18 months ago by seanfarley (Sean Farley)

Replying to iannucci@…:

Or we can just roll back. I just built and installed libedit from the portfile at rev. 127130, and it's working correctly with Python 3.4.

Any objections to this?

comment:59 Changed 18 months ago by eborisch (Eric A. Borisch)

This continues to cause problems:

Hearing no objections here in the past three months, I've committed the flip to readline in r145851 for python27 as a trial balloon. If we don't have an uproar (waiting for it) we can move over the others. You can install +libedit if you really want to, and once this is fixed, we can move the variant-less version back to libedit.

Note: I've left the port:libedit dependency to avoid the possibility of something that was built before and linked against libedit (indirectly "dependent" through python27) being broken by a 'port reclaim' operation.

comment:60 Changed 17 months ago by jmroot (Joshua Root)

What was wrong with recommending use of py*-readline when readline support is required? A variant is a worse solution since it requires a reinstall for everyone (even those who don't use python interactively) and it makes python effectively GPLv3.

comment:61 Changed 17 months ago by eborisch (Eric A. Borisch)

The problem is that we can't make python dependent (recursively) on py-readline, and it was installing broken software which prompted users to go find out what was going on and then try to fix it.

Yes, it requires a reinstall; for many users this will be a binary install (from packages.macports.org/python27). Still better than installing something that we've known has been broken for months and just figure users will search out the solution.

I'm all for moving it back to libedit once the combination is working. (Where working means not broken for interactive users.)

comment:62 Changed 17 months ago by ryandesign (Ryan Schmidt)

  • Cc brett@… added

Has duplicate #50794.

comment:63 follow-up: Changed 17 months ago by piannucci (Peter Iannucci)

Hm, perhaps I was too indirect. I object to flipping over to readline. I believe that there is already a good temporary solution -- roll back -- and a good long-term solution -- bisect and submit an upstream patch. In my view, flipping to readline would be akin to sweeping the problem under the rug.

comment:64 Changed 16 months ago by Russell-Jones-OxPhys (Russell Jones)

  • Cc russell.jones@… added

Cc Me!

comment:65 Changed 15 months ago by ryandesign (Ryan Schmidt)

  • Cc sporring@… added

Has duplicate #51188.

comment:66 follow-up: Changed 15 months ago by eborisch (Eric A. Borisch)

This is still causing problems, clearly. How about we add "notes" to all Python versions to "Please install pyNN-readline if you will be using Python interactively (from the command line)." Would that make everyone happy -- and hopefully help people get working software. Delivering broken tools that we know are broken is just bad form.

comment:67 in reply to: ↑ 66 Changed 15 months ago by roger@…

Replying to eborisch@…:

This is still causing problems, clearly. How about we add "notes" to all Python versions to "Please install pyNN-readline if you will be using Python interactively (from the command line)." Would that make everyone happy -- and hopefully help people get working software. Delivering broken tools that we know are broken is just bad form.

Would have made me happyier, more than hunting for this bug report.

Changed 15 months ago by eborisch (Eric A. Borisch)

comment:68 Changed 15 months ago by eborisch (Eric A. Borisch)

I plan to commit the python-notes.patch changes tomorrow unless someone objects for some bizarre reason. (There is no change to the installs, only an addition to notes to inform console users of issue / suggesting they install pyNN-readline.)

comment:69 Changed 15 months ago by eborisch (Eric A. Borisch)

Committed the notes patch (not a real solution, but at least a warning to users) in r148368.

comment:70 Changed 14 months ago by mojca (Mojca Miklavec)

  • Cc mojca@… added

Cc Me!

comment:71 in reply to: ↑ 63 Changed 14 months ago by ryandesign (Ryan Schmidt)

  • Cc ryandesign@… added

Replying to iannucci@…:

Hm, perhaps I was too indirect. I object to flipping over to readline. I believe that there is already a good temporary solution -- roll back -- and a good long-term solution -- bisect and submit an upstream patch. In my view, flipping to readline would be akin to sweeping the problem under the rug.

And causes new problems. Specifically, this change caused graphviz to no longer be considered distributable, because it eventually depends on python27, and readline's GPL license conflicts with graphviz's EPL license.

What is the status of getting this issue resolved "properly"? Is there an upstream bug report? (The only upstream bug report linked in this ticket so far is closed.)

Changed 13 months ago by eborisch (Eric A. Borisch)

Patch against r148368; reverts to libedit but leaves notes (install py27-readline suggestion) intact.

Changed 13 months ago by eborisch (Eric A. Borisch)

Reverse of upstream 1.111 changes.

comment:72 Changed 13 months ago by eborisch (Eric A. Borisch)

I've attached a patch that will make the python27 port act like the others, where it still installs and builds against libedit, but with the install time notes suggesting py27-readline be installed. I will commit it in a few days if there are no objections. This will "resolve" the licensing status, but means installing a broken (to my mind) python if the user runs sudo port install python27 and doesn't read/notice/act on the notes.

I've also attached a fix; but I'm not sure that it will be accepted upstream. Read on...

I believe I've found the (opposite) issue here: http://gnats.netbsd.org/48957. Arising from this PR, r1.111, undid a change in r1.85, which were from a set of Apple patches.

I can confirm that undoing the upstream (1.111) change (commenting out one line: readline_c.diff) fixes the broken shell state on python exit under macOS. This change, however, was clearly viewed as incorrect upstream.

For reference, Apple's version based on 1.106 currently has this line commented out.

I suggest we match Apple's libedit behavior here, at least for this quirk. Remove the patch if/when Apple aligns with NetBSD. (Add readline_c.diff as a patchfile.) At that time, changes may be required in Python, and appropriate upstream reports can be filed.

Alternatively, we could see if python's __APPLE__ fenced code in its own readline.c can be removed safely with the latest upstream libedit; I haven't had time to do this, but maybe someone else will...

As to how this impacts anything else using libedit, I can't say as I haven't looked.

comment:73 Changed 13 months ago by zaxdo@…

  • Cc zaxdo@… added

Cc Me!

comment:74 Changed 13 months ago by eborisch (Eric A. Borisch)

Committed changes to python27 in r149730. Builds broken (IMHO) against libedit, but hey, it's distributable so at least there is that. Note for the user to install py27-readline if they want to use the terminal remains.

I've seen no comments on the libedit readline.c patch I posted here. Perhaps I should make that into a separate issue...

comment:75 Changed 13 months ago by jmroot (Joshua Root)

The interesting part is that the problem didn't appear when using ncurses 5 with the same version of libedit. Ncurses 6 added its own buffering, whereas previously it shared buffers with stdio. The release notes mention that some programs may incorrectly assume that the buffers are the same. I wonder if python is doing some form of this.

I'd really prefer not to commit hacky workarounds that upstream says are incorrect, without understanding the problem. It might be instructive to file a radar asking Apple to adopt upstream's libedit changes?

comment:76 Changed 13 months ago by musicant@…

  • Cc musicant@… removed

Cc Me!

comment:77 Changed 13 months ago by musicant@…

  • Cc musicant@… added

Cc Me!

comment:78 Changed 13 months ago by musicant@…

  • Cc musicant@… removed

Cc Me!

comment:79 follow-ups: Changed 13 months ago by eborisch (Eric A. Borisch)

The new libedit (libedit-0160618-3.1_0) seems to have fixed this one. I can remove py27-readline, and things still work when I exit python in the terminal.

Can others confirm so we can finally put this out to pasture? I'll remove the notes from pythonXX after I see some "works for me" posts...

comment:80 in reply to: ↑ 79 ; follow-up: Changed 13 months ago by fhgwright (Fred Wright)

Replying to eborisch@…:

The new libedit (libedit-0160618-3.1_0) seems to have fixed this one. I can remove py27-readline, and things still work when I exit python in the terminal.

Can others confirm so we can finally put this out to pasture? I'll remove the notes from pythonXX after I see some "works for me" posts...

Still fails here (with python27 @2.7.12_1 and libedit @20160618-3.1_0+universal).

I wasn't sure of the relationship between py-readline and py27-readline, so I'd installed both, and uninstalled both to test this. I'd *also* previously installed python27 with the readline variant, which may have been belt-and-suspenders when combined with the separate package, but I switched to the default variants for the test.

Note that in the failing case, prompts aren't displayed properly in interactive use, as well as leaving the terminal screwed up on exit (though recoverable via "stty sane"). So the issue description is incomplete.

comment:81 in reply to: ↑ 79 Changed 13 months ago by Themanwithoutaplan

Replying to eborisch@…:

The new libedit (libedit-0160618-3.1_0) seems to have fixed this one. I can remove py27-readline, and things still work when I exit python in the terminal.

Can others confirm so we can finally put this out to pasture? I'll remove the notes from pythonXX after I see some "works for me" posts...

Sorry, to be pedantic but can you please be specific about what we should test? For me the latest Python 2.7 (2.7.12_1) and libedit (20160618-3.1_0) does not resolve the problem and python-readline is still required. Should I be working with a checkout? Also, doesn't this have as much to do with ncurses as it does with libedit?

comment:82 follow-up: Changed 13 months ago by eborisch (Eric A. Borisch)

Hrmmm. Looks like python needs to be rebuilt against the new headers. I'll revbump pythonXX assuming this pans out.

(replace 27 with the version you would like to test)

sudo port uninstall -f python27 py27-readline
sudo port upgrade --no-rev-upgrade libedit
sudo port install -s python27  # -s : source-only mode

comment:83 in reply to: ↑ 82 Changed 13 months ago by Themanwithoutaplan

Compiling from source does indeed solve the problem. Thanks!

comment:84 in reply to: ↑ 80 ; follow-up: Changed 13 months ago by mf2k (Frank Schima)

Replying to fw@…:

I wasn't sure of the relationship between py-readline and py27-readline

py-readline does nothing because it is a stub port that holds the real python modules such as py27-readline. This is true for all python modules in Macports. So py-* ports should never be installed.

comment:85 in reply to: ↑ 84 Changed 13 months ago by fhgwright (Fred Wright)

Replying to mf2k@…:

Replying to fw@…:

I wasn't sure of the relationship between py-readline and py27-readline

py-readline does nothing because it is a stub port that holds the real python modules such as py27-readline. This is true for all python modules in Macports. So py-* ports should never be installed.

Yeah, I suspected as much, but the fact that it installed without error obfuscated the issue.

I also tried as many other versions of Python as possible, in some cases requiring the expansion of the version lists in py-readline and py-setuptools. In one case (python31), the failure mode is completely different but the py-readline fix still works. All other versions from 26 through 35 behave identically. The python25 build is currently broken for other reasons, so I couldn't test that. The python24 port was never patched to use libedit in the first place, so it doesn't have the issue.

comment:86 Changed 13 months ago by larryv (Lawrence Velázquez)

  • Cc larryv@… added

Cc Me!

comment:87 Changed 13 months ago by eborisch (Eric A. Borisch)

False alarm. The latest pythons are tripping during build if readline is installed, leading to the readline module not being built, but the build still "succeeds."

From the build log when port:readline is installed:

Failed to build these modules:
readline

The resulting python doesn't hose the terminal as it doesn't have the libedit-backed readline module.

Installing pyXX-readline is still the best option for most users, followed by pythonXX +readline, with the latter requiring a ~ 10x longer compile. Both require local compilation.

comment:88 follow-up: Changed 10 months ago by icemac (Michael Howitz)

Installing python36 (@3.6.0b1_0) suggests to install py36-readline. But this port does not exist.

Does this problem no longer exist for Python 3.6 or should I file a new ticket that the port py36-readline needs to be created?

comment:89 Changed 10 months ago by icemac (Michael Howitz)

  • Cc mh@… removed
Last edited 10 months ago by icemac (Michael Howitz) (previous) (diff)

comment:90 Changed 10 months ago by icemac (Michael Howitz)

  • Cc mh@… added
Last edited 10 months ago by icemac (Michael Howitz) (previous) (diff)

comment:91 in reply to: ↑ 88 ; follow-up: Changed 10 months ago by larryv (Lawrence Velázquez)

We generally don’t add modules for new Python versions until they’re released.

However, do you observe the problem with 3.6.0 beta 1?

comment:92 in reply to: ↑ 91 Changed 10 months ago by icemac (Michael Howitz)

Replying to larryv@…:

However, do you observe the problem with 3.6.0 beta 1?

Yes, 3.6.0b1 has this problem, too. (I should have tried it before, sorry.)

Installing it with +readline fixes the problem and does not show the suggestion to install py36-readline.

comment:93 Changed 8 months ago by dsedivec

  • Cc dsedivec added

comment:94 follow-up: Changed 7 months ago by cdeil (Christoph Deil)

With sudo port install python36 I now see

##############################################################
# IF YOU ARE USING PYTHON FROM THE TERMINAL, PLEASE INSTALL:
#   py36-readline
# TO AVOID A LIBEDIT / PYTHON INTERACTION ISSUE.
# REF: https://trac.macports.org/ticket/48807
##############################################################

but py36-readline doesn't exist (yet?).

comment:95 in reply to: ↑ 94 ; follow-up: Changed 7 months ago by fhgwright (Fred Wright)

Replying to cdeil:

With sudo port install python36 I now see

##############################################################
# IF YOU ARE USING PYTHON FROM THE TERMINAL, PLEASE INSTALL:
#   py36-readline
# TO AVOID A LIBEDIT / PYTHON INTERACTION ISSUE.
# REF: https://trac.macports.org/ticket/48807
##############################################################

but py36-readline doesn't exist (yet?).

Note that Python 3.6 was only officially released two days ago :-)

And unfortunately it's not just a matter of adding the version to the py-readline port.

With a (locally generated) py36-readline installed:

MacPro:~ fw$ python3.6
Python 3.6.0 (default, Dec 23 2016, 12:59:30) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
Python(59962,0x7fff7a4d0310) malloc: *** error for object 0x103e7f640: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

Without it:

MacPro:~ fw$ python3.6
Python 3.6.0 (default, Dec 23 2016, 12:59:30) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
>>> 2
^D>>> 
MacPro:~ fw$ MacPro:~ fw$ MacPro:~ fw$ 

The latter shows the usual trouble until "stty sane" (unechoed).

comment:96 follow-up: Changed 7 months ago by eborisch (Eric A. Borisch)

You can install python36 +readline (python36 with the readline variant). I just tried it and it "worked for me."

comment:97 in reply to: ↑ 96 Changed 7 months ago by fhgwright (Fred Wright)

Replying to eborisch:

You can install python36 +readline (python36 with the readline variant). I just tried it and it "worked for me."

Yeah, that works. I'd moved away from that approach once py-readline appeared, since it avoids the need to rebuild Python from source due to the non-default variant. What's weird about this (aside from how long libedit's been broken) is that upstream Python uses readline and has to be patched to use libedit, so apparently having a less restrictive license was prioritized over consistency with upstream.

It might be nice if the readline variant didn't suggest installing pyXX-readline. :-)

comment:98 Changed 7 months ago by cdeil (Christoph Deil)

You can install python36 +readline (python36 with the readline variant). I just tried it and it "worked for me."

Yeah, that works. I'd moved away from that approach once py-readline appeared, since it avoids the need to rebuild Python from source due to the non-default variant.

+1 to add py36-readline, it's more convenient / faster than the +readline variant.

Most users (like me) will run into this issue after they already installed python36 and just being able to install py36-readline to fix it is the simplest solution (besides changing the install to not have this bug by default in the first place).

Please ...

comment:99 Changed 7 months ago by suominen (Kimmo Suominen)

  • Cc suominen added

comment:100 Changed 7 months ago by cdeil (Christoph Deil)

py36-readline was now added: #53243

But it results in Python REPL crash on exit, so it's not a solution.

Installing py36-gnureadline doesn't solve this "messed up terminal state on exit" issue.

Sigh.

Last edited 7 months ago by ryandesign (Ryan Schmidt) (previous) (diff)

comment:101 Changed 7 months ago by raimue (Rainer Müller)

Linking my findings from last April: https://lists.macosforge.org/pipermail/macports-dev/2016-April/032891.html

The problem appears to be that a bug in libedit that was fixed, but CPython actually relied on this bug to work properly. If this bug does not occur with CPython on other platforms using libedit, it might be worth a shot to scrub the whole #ifdef __APPLE__ parts for readline/libedit handling. This use of #ifdef __APPLE__ seems to be meant as "using libedit as shipped by Apple", but this is clearly wrong and a bad way to match this condition.

comment:102 Changed 6 months ago by rogererens (Roger Erens)

  • Cc rogererens added

comment:103 Changed 6 months ago by lesinigo (Luca Lesinigo)

  • Cc lesinigo added

comment:104 Changed 6 months ago by dsedivec

  • Cc dsedivec removed

comment:105 Changed 6 months ago by dsedivec

  • Cc dsedivec added

comment:106 Changed 5 months ago by aikchar (Hamza Sheikh)

  • Cc aikchar added

comment:107 Changed 5 months ago by mohd-akram (Mohamed Akram)

I don't have this issue when I build python36 myself on macOS 10.12.3:

sudo port -s install python36

Packages need an update?

comment:108 Changed 5 months ago by mohd-akram (Mohamed Akram)

  • Cc mohd-akram added

comment:109 Changed 5 months ago by eborisch (Eric A. Borisch)

2 questions:

  • Do you have py36-readline installed?
  • What shell are you using?

comment:110 follow-up: Changed 5 months ago by mohd-akram (Mohamed Akram)

I don't have py36-readline installed and I'm using the default bash shell.

comment:111 in reply to: ↑ 110 ; follow-up: Changed 5 months ago by eborisch (Eric A. Borisch)

I'm guessing you had the readline port installed while you built python3.6. Try building in trace/sandbox mode. sudo port uninstall python36 && sudo port -ts install python36. By my brief test, this will yield a broken (as measured by having a bad terminal state on exit) python3.6.

Please confirm, and assuming you see the same results, we can try to track down why having readline present -- even just during configure; you can remove it between configure and the remainder of the install process -- on the system makes this problem go away.

comment:112 Changed 5 months ago by mohd-akram (Mohamed Akram)

Indeed, it was because I had readline installed.

Changed 5 months ago by mohd-akram (Mohamed Akram)

Updated libedit patch

comment:113 in reply to: ↑ 111 Changed 5 months ago by raimue (Rainer Müller)

Replying to eborisch:

Please confirm, and assuming you see the same results, we can try to track down why having readline present -- even just during configure; you can remove it between configure and the remainder of the install process -- on the system makes this problem go away.

The existing patch-libedit.diff replaces -lreadline with -ledit, which makes the check for readline to actually link with the readline compatibility layer of libedit. However, there are other checks still using -lreadline in the configure script.

@mohd-akram Regarding the attached patch, these hunks in your patch look bogus:

-#ifdef __APPLE__
+#ifndef __APPLE__

As I understand it, these #ifdef blocks were meant specifically for libedit as provided by Apple on macOS. Negating the conditional does not make sense to me, as that would mean they should be applied to all other systems. Just for additional background, most of this was added as a workaround to detect a possible bug in libedit as distributed by Apple at runtime. In order to eventually get such a fix upstream, the logic should be precise. Or just delete the conditional code with the patch, but then we will never get this upstream.

If you want to get rid of these blocks completely, the right thing would be to add an additional configure check defining a macro indicating whether it is using the broken Apple libedit or the fixed version from MacPorts.

comment:114 Changed 5 months ago by mohd-akram (Mohamed Akram)

Yeah, the patch is no good, I'll have to look into this more. There are readline checks all over the place in configure and the readline.c file. Someone was working to push this upstream with a proper fix - https://bugs.python.org/issue13501 - which might be useful.

comment:115 Changed 5 months ago by reubano (Reuben Cummings)

  • Cc reubano added

comment:116 in reply to: ↑ 95 Changed 4 months ago by lpsinger (Leo Singer)

Replying to fhgwright:

Replying to cdeil: With a (locally generated) py36-readline installed:

MacPro:~ fw$ python3.6
Python 3.6.0 (default, Dec 23 2016, 12:59:30) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
Python(59962,0x7fff7a4d0310) malloc: *** error for object 0x103e7f640: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

I get this crash too.

comment:117 Changed 4 months ago by davidchambers (David Chambers)

  • Cc davidchambers added

comment:118 follow-up: Changed 4 months ago by joshenders (Josh Enders)

I get this crash as well:

$ sudo port install python36 +readline
--->  Computing dependencies for python36
--->  Fetching archive for python36
--->  Attempting to fetch python36-3.6.1_0+readline.darwin_16.x86_64.tbz2 from https://packages.macports.org/python36
--->  Attempting to fetch python36-3.6.1_0+readline.darwin_16.x86_64.tbz2 from http://sea.us.packages.macports.org/macports/packages/python36
--->  Attempting to fetch python36-3.6.1_0+readline.darwin_16.x86_64.tbz2 from http://kmq.jp.packages.macports.org/python36
--->  Fetching distfiles for python36
--->  Attempting to fetch Python-3.6.1.tar.xz from https://distfiles.macports.org/python36
--->  Attempting to fetch Python-3.6.1.tar.xz from http://www.python.org/ftp/python/3.6.1/
--->  Verifying checksums for python36
--->  Extracting python36
--->  Applying patches to python36
Warning: reinplace s|xargs -0 rm -r|/usr/bin/xargs -0 /bin/rm -r|g didn't change anything in /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_python36/python36/work/Python-3.6.1/Mac/PythonLauncher/Makefile.in
--->  Configuring python36
--->  Building python36
--->  Staging python36 into destroot
--->  Installing python36 @3.6.1_0+readline
--->  Deactivating python36 @3.6.1_0
--->  Cleaning python36
--->  Activating python36 @3.6.1_0+readline
--->  Cleaning python36
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  Some of the ports you installed have notes:
  python36 has the following notes:
    To make this the default Python or Python 3 (i.e., the version run by the 'python' or 'python3' commands), run one or both of:

        sudo port select --set python python36
        sudo port select --set python3 python36

    ##############################################################
    # IF YOU ARE USING PYTHON FROM THE TERMINAL, PLEASE INSTALL:
    #   py36-readline
    # TO AVOID A LIBEDIT / PYTHON INTERACTION ISSUE.
    # REF: https://trac.macports.org/ticket/48807
    ##############################################################

$ python3
Python 3.6.1 (default, Mar 24 2017, 00:08:29)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello World")
Python(61973,0x7fffaf4293c0) malloc: *** error for object 0x10f92f300: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

comment:119 in reply to: ↑ 118 ; follow-up: Changed 4 months ago by eborisch (Eric A. Borisch)

Replying to joshenders:

I get this crash as well:

[snip]

---> Activating python36 @3.6.1_0+readline

Do you also have py36-readline installed? Having both python36 (itself) installed +readline (variant) and py36-readline (the readline python package) installed at the same time can cause that, if I recall. Use one or the other.

comment:120 in reply to: ↑ 119 Changed 4 months ago by aikchar (Hamza Sheikh)

Replying to eborisch:

Replying to joshenders:

I get this crash as well:

[snip]

---> Activating python36 @3.6.1_0+readline

Do you also have py36-readline installed? Having both python36 (itself) installed +readline (variant) and py36-readline (the readline python package) installed at the same time can cause that, if I recall. Use one or the other.

I removed py36-readline and installed python36 +readline and I don't get this error anymore. Thanks, Eric!

$ port installed | grep readline
  py27-readline @6.2.4.1_1 (active)
  py34-readline @6.2.4.1_1 (active)
  py35-readline @6.2.4.1_1 (active)
  readline @6.3.003_1 (active)

$ sudo port install python36 +readline
<SNIP OUTPUT>

$ port installed | grep readline
  py27-readline @6.2.4.1_1 (active)
  py34-readline @6.2.4.1_1 (active)
  py35-readline @6.2.4.1_1 (active)
  python36 @3.6.1_0+readline (active)
  readline @6.3.003_1 (active)

$ /opt/local/bin/python3.6
Python 3.6.1 (default, Mar 24 2017, 15:43:12)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello, World!")
Hello, World!
>>> exit()

$ echo $?
0

$ sudo port install py36-readline
<SNIP OUTPUT>

$ port installed | grep readline
  py27-readline @6.2.4.1_1 (active)
  py34-readline @6.2.4.1_1 (active)
  py35-readline @6.2.4.1_1 (active)
  py36-readline @6.2.4.1_1 (active)
  python36 @3.6.1_0+readline (active)
  readline @6.3.003_1 (active)

$ /opt/local/bin/python3.6
Python 3.6.1 (default, Mar 24 2017, 15:43:12)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello, World!")
Python(58844,0x7fffd2b4a3c0) malloc: *** error for object 0x1041d6318: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

$ echo $?
134

$ sudo port uninstall py36-readline
<SNIP OUTPUT>

$ port installed | grep readline
  py27-readline @6.2.4.1_1 (active)
  py34-readline @6.2.4.1_1 (active)
  py35-readline @6.2.4.1_1 (active)
  python36 @3.6.1_0+readline (active)
  readline @6.3.003_1 (active)

$ /opt/local/bin/python3.6
Python 3.6.1 (default, Mar 24 2017, 15:43:12)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello, World!")
Hello, World!
>>> exit()

$ echo $?
0

comment:121 Changed 3 months ago by bs (Britt Selvitelle)

Just installed python36 and the install recommend py36-readline and had malloc crashes along the lines of pointer being freed was not allocated python.

Found this thread. Uninstalled py36-readline and installed python36 +readline, which seems to have fixed this. Can this be pushed as a default flag, at the very least in place of the current misinformation that py36-readline works?

comment:122 Changed 3 months ago by 1-61803

  • Cc 1-61803 added

comment:123 Changed 2 months ago by aminggs (Andrew Inggs)

  • Cc aminggs added

comment:124 Changed 2 months ago by michaellass (Michael Lass)

  • Cc michaellass added

comment:125 Changed 2 months ago by stromnov (Andrew Stromnov)

  • Cc stromnov added

comment:126 Changed 2 months ago by ibrahimkarahan

  • Cc ibrahimkarahan added

comment:127 Changed 8 weeks ago by jamadden (Jason Madden)

  • Cc jamadden added
Note: See TracTickets for help on using tickets.