python26 does not provide _bsddb

[guy.linton@…]

I have built python26, but it does not find the _bsddb module:

[Thu Apr 01 10:19:56]G5:~ tim$ sudo python_select python26
Password:
Selecting version "python26" for python
[Thu Apr 01 10:20:36]G5:~ tim$ python
Python 2.6.5 (r265:79063, Mar 21 2010, 12:06:08) 
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bsddb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/bsddb/__init__.py", line 64, in <module>
    import _bsddb
ImportError: No module named _bsddb

I think that perhaps bdsdb for python26 is in bsddb185.so instead of _bsddb.so. But in my build, it does not seem to be complete. The last bit of otool -TMvDlL for python25 _bsddb.so says:

Load command 3
          cmd LC_LOAD_DYLIB
      cmdsize 60
         name /opt/local/lib/db46/libdb-4.6.dylib (offset 24)
   time stamp 1269169817 Sun Mar 21 11:10:17 2010
      current version 0.0.0
compatibility version 0.0.0

...skipped

Load command 7
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 8010
     iextdefsym 8010
     nextdefsym 5
      iundefsym 8015
      nundefsym 72
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 194972
  nindirectsyms 134
      extreloff 0
        nextrel 0
      locreloff 94208
        nlocrel 465
        /opt/local/lib/db46/libdb-4.6.dylib (compatibility version 0.0.0, current version 0.0.0)
        time stamp 1269169817 Sun Mar 21 11:10:17 2010
        /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
        time stamp 1159409917 Thu Sep 28 03:18:37 2006
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.12)
        time stamp 1264792999 Fri Jan 29 19:23:19 2010

but for python26, the last part of the otool output is:

Load command 6
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 1104
     iextdefsym 1104
     nextdefsym 1
      iundefsym 1105
      nundefsym 41
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 30680
  nindirectsyms 75
      extreloff 0
        nextrel 0
      locreloff 16384
        nlocrel 68
        /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
        time stamp 1159409917 Thu Sep 28 03:18:37 2006
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.12)
        time stamp 1264792999 Fri Jan 29 19:23:19 2010

Please let me know what flags for otool are needed if you want the rest of the otool output.

[macsforever2000@…]

FYI, it works fine for me. I'm on Intel Snow Leopard. What OS version and architecture are you running? It looks like you are on 10.5 right?

[guy.linton@…]

I'm on Mac OS X 10.4.11 on PowerPC G5.

[blb@…]

How did you build db46, python25, and python26?

$ port installed db46 python25 python26

Usually when bsddb (or a few other modules) have issues, there's a problem linking to the dependents (then, unfortunately, python continues to build and simply doesn't install a working module).

[guy.linton@…]

I suppose you mean port install.. not installed. I just did

sudo port build <pointing at a local copy of gramps Portfile>

This builds in the order db46 python26 python25. Could the problem be that I have built python26 before python25? Should python26 be dependent on python25, or is there some other dependency missing from python26? Please note that my build was starting from a completely fresh installation of MacPorts with nothing built previously. Can you tell me what you get for otool -TMvDlL bsddb185.so.

List of build order:

--->  Building expat
--->  Building gperf
--->  Building libiconv
--->  Building ncursesw
--->  Building ncurses
--->  Building gettext
--->  Building lzmautils
--->  Building texinfo
--->  Building aspell
--->  Building aspell-dict-en
--->  Building xmlcatmgr
--->  Building docbook-xml-4.1.2
--->  Building docbook-xml-4.2
--->  Building docbook-xml-4.3
--->  Building docbook-xml-4.4
--->  Building docbook-xml-4.5
--->  Building docbook-xml-5.0
--->  Building docbook-xml
--->  Building docbook-xsl
--->  Building iso-codes
--->  Building zlib
--->  Building libxml2
--->  Building libxslt
--->  Building perl5.8
--->  Building perl5
--->  Building p5-xml-parser
--->  Building pkgconfig
--->  Building bzip2
--->  Building db46
--->  Building gdbm
--->  Building openssl
--->  Building readline
--->  Building sqlite3
--->  Building freetype
--->  Building fontconfig
--->  Building xorg-bigreqsproto
--->  Building xorg-inputproto
--->  Building xorg-kbproto
--->  Building xorg-xproto
--->  Building xorg-libXau
--->  Building xorg-libXdmcp
--->  Building xorg-util-macros
--->  Building xorg-xcmiscproto
--->  Building xorg-xextproto
--->  Building xorg-xf86bigfontproto
--->  Building xorg-xtrans
--->  Building xorg-libX11
--->  Building xorg-renderproto
--->  Building xrender
--->  Building Xft2
--->  Building tcl
--->  Building xorg-libXext
--->  Building xorg-scrnsaverproto
--->  Building xorg-libXScrnSaver
--->  Building tk
--->  Building python26
--->  Building py26-libxml2
--->  Building getopt
--->  Building gnome-common
--->  Building p5-getopt-long
--->  Building p5-pathtools
--->  Building p5-scalar-list-utils
--->  Building intltool
--->  Building rarian
--->  Building gnome-doc-utils
--->  Building gtk-doc
--->  Building p5-locale-gettext
--->  Building help2man
--->  Building m4
--->  Building autoconf
--->  Building automake
--->  Building libtool
--->  Building glib2
--->  Building atk
--->  Building libpixman
--->  Building libpng
--->  Building cairo
--->  Building jpeg
--->  Building jasper
--->  Building pango
--->  Building shared-mime-info
--->  Building tiff
--->  Building xorg-compositeproto
--->  Building xorg-fixesproto
--->  Building xorg-libXfixes
--->  Building xorg-libXcomposite
--->  Building xorg-libXcursor
--->  Building xorg-damageproto
--->  Building xorg-libXdamage
--->  Building xorg-libXi
--->  Building xorg-xineramaproto
--->  Building xorg-libXinerama
--->  Building xorg-randrproto
--->  Building xorg-libXrandr
--->  Building gtk2
--->  Building libglade2
--->  Building gdl
--->  Building db47
--->  Building flex
--->  Building dbus
--->  Building dbus-glib
--->  Building popt
--->  Building desktop-file-utils
--->  Building libidl
--->  Building orbit2
--->  Building policykit
--->  Building gconf
--->  Building gnome-mime-data
--->  Building gnome-vfs
--->  Building libgpg-error
--->  Building libgcrypt
--->  Building libtasn1
--->  Building gnome-keyring
--->  Building lzo
--->  Building opencdk
--->  Building gnutls
--->  Building xorg-libice
--->  Building xorg-libsm
--->  Building xorg-libXt
--->  Building xorg-libXmu
--->  Building libproxy
--->  Building libsoup
--->  Building libgda4
--->  Building libgtkhtml
--->  Building libbonobo
--->  Building audiofile
--->  Building esound
--->  Building libgnome
--->  Building libart_lgpl
--->  Building libgnomecanvas
--->  Building libbonoboui
--->  Building hicolor-icon-theme
--->  Building p5-xml-namespacesupport
--->  Building p5-xml-sax
--->  Building p5-xml-simple
--->  Building icon-naming-utils
--->  Building gnome-icon-theme
--->  Building libgnomeui
--->  Building gmp
--->  Building mpfr
--->  Building gcc43
--->  Building gzip
--->  Building atlas
--->  Building fftw-3
--->  Building python25
--->  Building py25-setuptools
--->  Building py25-nose
--->  Building py25-numpy
--->  Building py25-cairo
--->  Building py25-gobject
--->  Building py25-gtk
--->  Building py25-orbit
--->  Building py25-gnome
--->  Building gnome-python-extras
--->  Building hunspell
--->  Building enchant
--->  Building gtkspell2
--->  Building py25-bsddb
--->  Building py25-hashlib
--->  Building py25-zlib
--->  Building gramps
--->  Building python_select

[blb@…]

Replying to guy.linton@…:

I suppose you mean port install.. not installed. I just did

I actually did mean installed; when you run it, you'll see (it shows how those ports are installed).

[guy.linton@…]

Sorry, The answer is:

The following ports are currently installed:
  db46 @4.6.21_6 (active)
  python25 @2.5.5_0+darwin+darwin_8+macosx (active)
  python26 @2.6.5_0+darwin (active)

[blb@…]

Okay, no universal stuff going on. What is your Xcode version, as older gcc versions shipped with older Xcodes have caused issues? 2.5 is latest/last for 10.4 (see the FAQ for info on upgrading if you have an older one).

[guy.linton@…]

Yes, just upgraded because some component demanded a newer version - I have Xcode 2.5.

[blb@…]

Was that after you installed the python26 port (check the date of /opt/local/var/macports/receipts/python26/2.6.5_0+darwin/receipt.bz2 to see when it was installed). If the Xcode upgrade happened after that, then definitely try uninstalling/reinstalling the python26 port.

If, however, if was before then I take it the file /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_bsddb.so does not exist? If it doesn't, what is the output from

$ grep DB_VERSION_STRING /opt/local/include/db46/db.h

[guy.linton@…]

The upgrade to Xcode was AFTER I installed the python26 port.

python26 was installed on 21 March, but the upgrade to Xcode was 22 March when I had "libtheora 1.1.1 requires Xcode 2.5 or later but you have Xcode 2.4.1." (Xcode modification date is 22 Mar 14:01, while the python26 receipt is 21 Mar 12:11).

I will try uninstalling/reinstalling python26.

[The file /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_bsddb.so does NOT exist. The output from the grep is #define DB_VERSION_STRING "Berkeley DB 4.6.21: (September 27, 2007)"]

[guy.linton@…]

I have now done

sudo port -f uninstall python26
sudo port install python26
sudo python_select python26
python /opt/local/share/gramps/gramps.py

Gramps starts OK, but when it tries to load the most recent database, it complains about the format of the Berkley database (Database needs recovery, cannot open it!

Low level database corruption detected
Gramps has detected a problem in the underlying Berkeley
database. This can be repaired by from the Family Tree
Manager. Select the database and click on the Repair button

DB_RUNRECOVERY: Fatal error, run database recovery --
process-private: unable to find environment

After supposedly repairing the database, when I then try to open it I get an"Unexpected Error":

86421: ERROR: gramps.py: line 138: Unhandled exception
Traceback (most recent call last):
  File "/opt/local/share/gramps/gui/dbman.py", line 649, in __repair_db
    restore(dbase)
  File "/opt/local/share/gramps/gen/db/backup.py", line 153, in restore
    __do_restore(database)
  File "/opt/local/share/gramps/gen/db/backup.py", line 170, in __do_restore
    database.rebuild_secondary()
  File "/opt/local/share/gramps/gen/db/write.py", line 222, in try_
    raise Errors.DbError(msg)
DbError: Invalid argument -- /Users/tim/.gramps/grampsdb/4b229499/person_id.db: unexpected file type or format

So, it seems that although the database is found OK, there is something seriously wrong with the database build as found by python 2.6 (or at least it is not compatible with that from python 2.5).

[guy.linton@…]

I should have said that after doing the reinstall, /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_bsddb.so DOES exist.

[blb@…]

Replying to guy.linton@…:

I should have said that after doing the reinstall, /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_bsddb.so DOES exist.

Okay, that suggests that Xcode 2.4.1 had some problem with building the bsddb module that 2.5 fixed; I guess we can call that the good news.

As far as the database incompatibilities, that's definitely weird; python25 uses the same version of BDB (db46) as python26 so you'd think they would be compatible.

There is a  gramps bug that reads similar to your issue. Perhaps the steps there (using db46_recover which should be installed with the db46 port) may help?

[guy.linton@…]

As you say, the bsddb module now seems to be available to python26.

As you say, the problem is weird. The failure is reported here:

        try:
            try:
                self.dbstate.db.load(filename, self._pulse_progress, 
                                     mode, upgrade=False)
            except gen.db.exceptions.DbUpgradeRequiredError, msg:
                if QuestionDialog2(_("Need to upgrade database!"), 
                                   str(msg), 
                                   _("Upgrade now"), 
                                   _("Cancel")).run():
                    self.dbstate.db.load(filename, self._pulse_progress, 
                                         mode, upgrade=True)
                    self.dbstate.db.set_save_path(filename)
                else:
                    self.dbstate.no_database()
        except gen.db.exceptions.DbVersionError, msg:
            self.dbstate.no_database()
            self._errordialog( _("Cannot open database"), str(msg))
        except OSError, msg:
            self.dbstate.no_database()
            self._errordialog(
                _("Could not open file: %s") % filename, str(msg))
        except Errors.DbError, msg:
            self.dbstate.no_database()
            self._dberrordialog(msg)
        except Exception:
            self.dbstate.no_database()

Where _dberrordialog is being called. The msg attached is:

DB_RUNRECOVERY: Fatal error, run database recovery -- process-private: unable to find environment

It doesn't seem to be that the database is corrupt, but rather that something is wrong with the 'environment'. Do you have any ideas what dbload may be trying to say. What is this environment it is trying to find?

db46_recover doesn't seem to make any difference.

There are known incompatibilities between the databases in Gramps 3.1 and Gramps 3.2. These are due to a mistake in Gramps about block size (or something like that). The work-around is to export the database from 3.1 (in a Gramps format) and then import into 3.2. The problem has now been fixed in a forthcoming release of Gramps.

So, I have tried importing the Gramps dump into a fresh python26 database, and that works OK. Switching back to python25 gives a message about database corruption (not the same environment message).

However, when I run Gramps with python26, I get the slow running reported in  http://www.gramps-project.org/bugs/view.php?id=3750. I don't think the switch to the more recent database is possible because there is no Macports version of it.

There is clearly a difference between the behaviour of

5316156  520 -rwxr-xr-x    2 root     wheel      263404 Mar 21 22:12 ./local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload/_bsddb.so
6113516  640 -rwxr-xr-x    2 root     wheel      325416 Apr  8 22:44 ./local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_bsddb.so

In conclusion (1) Any idea how I could find out what the problem with switching between the two different implementations of _bsddb.so is? (At least in one direction, it seems to be that the location of the environment is lost). (2) The python 2.6 version of the database is slow.

[blb@…]

Sounds as if the issue may be with python 2.6's bsddb, at least reading over the gramps bug, right? Has anyone filed a  bug with python? I didn't see anything with a quick search; could be helpful.

The MacPorts build shouldn't be doing anything weird with that module (you can see all the patches which go into the port in trac as well as the Portfile itself. Nothing there is changing bsddb-specific functionality or patching, as far as I can tell.

[guy.linton@…]

Replying to blb@…:

Sounds as if the issue may be with python 2.6's bsddb, at least reading over the gramps bug, right? Has anyone filed a  bug with python?

I have raised  a bug with python "issue8504 bsddb databases in python 2.6 are not compatible with python 2.5 and are slow in python 2.6" for this.

[elelay@…]

hi, jumping into this because I would like to test gramp...

I can't have the _bsddb.so module built with python26 @2.6.6_0+no_tkinter. I'm on 10.4.11 ppc G4 XCode 2.5 and uninstalled db46 python26 and then reinstalled this afternoon, with no success.

running sudo port -d destroot python26, this is what I found :

Failed to find the necessary bits to build these modules:
_bsddb             linuxaudiodev      ossaudiodev     
spwd               sunaudiodev                        
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

So bsddb46 is not found. To my mind, it is due to  this change, because in the list of working architectures there isn't "Power Macintosh", so setup.py ignores the installed db46. The change dates back to april, so it wasn't there in the version of python Guy tried. Overwise I don't understand how he managed to build _bsddb.so

Indeed, when I edit setup.py line 805 to add 'Power Macintosh' like this :

                if arch not in ('i386', 'i486', 'i586', 'i686',
                                'x86_64', 'ia64', 'Power Macintosh'):

_bsddb.so is built. I'll test some more and provide a patch.

[elelay@…]

Cc Me!

[elelay@…]

diff against 72533

[elelay@…]

with this patch, _bsdb is built and working.

[elelay@…]

the upstream bug has been closed as invalid. applying the patch in r72668. To sum-up, the issue in python26@2.6.6_0 is that python rejects db46 on ppc because it's allegedly unstable, so the bsddb module is not built.