Changes between Version 222 and Version 223 of SummerOfCode


Ignore:
Timestamp:
Feb 3, 2014, 10:36:34 PM (10 years ago)
Author:
neverpanic (Clemens Lang)
Comment:

Go over all proposals and refactor

Legend:

Unmodified
Added
Removed
Modified
  • SummerOfCode

    v222 v223  
    9898* Potential mentors: cal
    9999
    100 ==== Remove dependency on Xcode ==== #xcode
    101 
    102 MacPorts currently requires a full Xcode installation, even if the user
    103 only wishes to install prepackaged binaries or compile ports that do not
    104 use Xcode projects for building. This task be as simple as allowing
    105 MacPorts to install binary packages without Xcode present, or as complex
    106 as enabling MacPorts to be completely self-sufficient (not requiring
    107 Xcode or the Command Line Tools at all).
    108 
    109 * Difficulty: Medium to very challenging
    110 * Languages: Tcl, C
    111 * Potential mentors: TBD
     100==== Phase out dependency on Xcode ==== #xcode
     101
     102MacPorts currently requires a full Xcode installation, even though a lot of ports will install just fine with the Command Line Tools package only. Since we also have a number of ports that need Xcode to build, we cannot completely remove the Xcode dependency. Your task would be to provide a way for maintainers to easily identify ports that depend on Xcode and mark them as such, so MacPorts can warn users without Xcode installed that a port they want to install needs the full Xcode package.
     103
     104To achieve this, you can modify "trace mode", a `DYLD_INSERT_LIBRARIES`-based sandbox to track whether a port has accessed files belonging to the Xcode package. If it does, your modifications should cause a warning to be printed suggesting the port maintainers to add `use_xcode yes` to the Portfile (unless of course, it is already there). You should also implement an error message if a user without Xcode installed tries to install a port that has `use_xcode yes` set.
     105
     106* Difficulty: Medium
     107* Languages: Tcl, C
     108* Potential mentors: cal
    112109
    113110==== MacPorts port for self-management ==== #self-management
    114111
    115 The MacPorts port should be the source for updating a user’s MacPorts
    116 installation.
    117 
    118 Currently the MacPorts port is used to build the .dmg installer for
    119 MacPorts that is used for the initial installation of MacPorts, and port
    120 uses the “selfupdate” mechanism for maintaining the MacPorts
    121 installation. The selfupdate mechanism is (at least not documented as
    122 such) not accessible through the MacPorts API and does not use the
    123 MacPorts mechanisms for maintaining ports.
     112The MacPorts port should be the source for updating a user’s MacPorts installation.
     113
     114Currently the MacPorts port is used to build the .dmg installer for MacPorts that is used for the initial installation of MacPorts, and port uses the “selfupdate” mechanism for maintaining the MacPorts installation. The selfupdate mechanism is (at least not documented as such) not accessible through the MacPorts API and does not use the MacPorts mechanisms for maintaining ports.
    124115
    125116* Difficulty: Challenging
     
    133124===== Perl modules integration from CPAN ===== #cpan2port
    134125
    135 There has been [[browser:contrib/cpan2port|an attempt]] to write
    136 a script for automatic generation of Portfiles from CPAN. This would
    137 simplify the maintenance of Perl modules in MacPorts. Revive this
    138 project and finish the script or rewrite it.
     126There has been [[browser:contrib/cpan2port|an attempt]] to write a script for automatic generation of Portfiles from CPAN. This would simplify the maintenance of Perl modules in MacPorts. Revive this project and finish the script or rewrite it.
    139127
    140128Resources:
     
    149137===== Read packages from Python's PIP (pip2port) ===== #pip2port
    150138
    151 MacPorts should be able to more readily install python packages from
    152 pip. We don't want to integrate the build/installation process of PIP,
    153 simply parse information or eggs it provides for various packages. This
    154 information will then be used to generate a Portfile, akin to
    155 [[browser:contrib/cpan2port/cpan2port|cpan2port]].
    156 
    157 There may be instances where MacPorts package names don't match up with
    158 pip’s. In this case, pip should be the authority and all improperly
    159 named MacPorts packages should be replaced by a renamed copy. Verbose
    160 automation of this step, akin to the portcheckup script, is ideal.
    161 
    162 The [[browser:contrib/portfile-gen/portfile-gen|portfile generator]]
    163 might be helpful.
     139MacPorts should be able to more readily install python packages from pip. We don't want to integrate the build/installation process of PIP, simply parse information or eggs it provides for various packages. This information will then be used to generate a Portfile, akin to [[browser:contrib/cpan2port/cpan2port|cpan2port]].
     140
     141There may be instances where MacPorts package names don't match up with pip’s. In this case, pip should be the authority and all improperly named MacPorts packages should be replaced by a renamed copy. Verbose automation of this step, akin to the portcheckup script, is ideal.
     142
     143The [[browser:contrib/portfile-gen/portfile-gen|portfile generator]] might be helpful.
    164144
    165145* Difficulty: Medium
     
    169149===== Read packages from other various package managers ===== #foo2port
    170150
    171 As above with the pip2port proposal, except with other package managers,
    172 such as [http://opam.ocamlpro.com/ opam] for ocaml packages,
    173 [http://www.haskell.org/cabal/ cabal] for haskell, [http://luarocks.org/ luarocks]
    174 for lua, [https://npmjs.org/ npm] for node.js, and so on.
     151As above with the pip2port proposal, except with other package managers, such as [http://opam.ocamlpro.com/ opam] for ocaml packages, [http://www.haskell.org/cabal/ cabal] for haskell, [http://luarocks.org/ luarocks] for lua, [https://npmjs.org/ npm] for node.js, and so on.
    175152
    176153* Classification: Medium
     
    182159To ease creation of new ports, the helper script [http://trac.macports.org/browser/contrib/portfile-gen/portfile-gen portfile-gen] is able to generated a Portfile stub by taking the name, version and possible a port group as input. This should be extended to handle more things automatically. Lots of options in a Portfile need to be figured out by the maintainer, although they could be extracted automatically from the source.
    183160
    184 With just a URL to the tarball of a software, portfile-gen can usually automatically extract the name and the version from the filename. Furthermore, the master_sites and distfiles can be derived from the URL, with special handling for sites like sourceforge/gnu/etc. However, it should also be possible to specify this later and start portfile-gen with a pre-downloaded tarball. The checksums for the distfiles can be generated after fetching the tarball (portfile authors should be warned to verify these with upstream!) and even more information can be collected when extracting the tarball.
     161With just a URL to the tarball of a software, portfile-gen can usually automatically extract the name and the version from the filename. Furthermore, the master_sites and distfiles can be derived from the URL, with special handling for sites like sourceforge/gnu/github/etc. However, it should also be possible to specify this later and start portfile-gen with a pre-downloaded tarball. The checksums for the distfiles can be generated after fetching the tarball (portfile authors should be warned to verify these with upstream!) and even more information can be collected when extracting the tarball.
    185162
    186163Different build systems usually have a unique way to be detected. For example, if a `configure.ac` or `configure.in` exists, but no `configure`, we need to run autoconf. If there is also a `Makefile.am`, but no `Makefile.in` this is using automake and we should better use `autoreconf` instead. If there is a `CMakeLists.txt`, we should include the cmake port group. If there is just a Makefile, but no `configure`, chances are high we can skip the configure phase with `use_configure no`.
     
    194171* Potential mentors: raimue
    195172
    196 ==== Increase test coverage ==== #testing
    197 
    198 MacPorts currently includes a [[source:trunk/base/tests|test framework]]
    199 to test features of the infrastructure. However, the tests do not cover
    200 all the code. This task consists of extending the test framework and
    201 could be broadened to develop a code coverage technology for MacPorts to
    202 make the infrastructure more robust to future changes.
    203 
    204 * Difficulty: Relatively easy to very challenging
    205 * Languages: Tcl, C
    206 * Potential mentor: raimue
    207 
    208173==== Configuration and environment selftest ==== #envcheck
    209174
    210 Add a command to check current setup for common pitfalls (e.g. stuff in
    211 `/usr/local`), inspired by Homebrew’s `brew doctor` functionality.
     175Add a command to check current setup for common pitfalls (e.g. stuff in `/usr/local`), inspired by Homebrew’s `brew doctor` functionality.
    212176
    213177* Difficulty: Easy
     
    215179* Potential mentors: larryv
    216180
    217 ==== Improve Trace mode ==== #tracemode
    218 
    219 Improve trace mode to the point where developers (and the build server)
    220 could run with it always turned on. This currently requires fixing some
    221 (minor) remaining bugs in the trace code itself, adjust the build system
    222 to always build the tracing library as universal binary and improving
    223 the Tcl backend of trace mode (e.g. by adding more features like a list
    224 of accessed files, speeding up the lookup by using better data
    225 structures).
     181==== Speed up trace mode ==== #tracemode
     182
     183Trace mode is a library preloading-based sandbox used to hide files that a port does not depend on or that are not part of a standard system's installation (such as `/usr/local`). This can avoid problems due to incompatible user-installed software and avoid "automagic" dependencies and increase the reproducibility of builds.
     184
     185Unfortunately, enabling trace mode adds a significant performance penalty to the build process. However, the trace mode code can certainly be optimized using appropriate cache data structures, such as a modified [http://en.wikipedia.org/wiki/Trie Trie]. Your task would be to identify the performance bottle necks, draft appropriate caching data structures and implement them.
    226186
    227187* Difficulty: Medium to Easy
     
    231191==== Improve startupitem code ==== #startupitem
    232192
    233 MacPorts has the ability to automatically generate startup items for the
    234 current platform. For OS X, these are plist files for launchd which will
    235 be installed as `/Library/LaunchDaemons/org.macports.*.plist`. The
    236 current code would need a little care and could make use of options
    237 which have been added in recent releases of launchd.
     193MacPorts has the ability to automatically generate startup items for the current platform. For OS X, these are plist files for launchd which will be installed as `/Library/LaunchDaemons/org.macports.*.plist`. The current code would need a little care and could make use of options which have been added in recent releases of launchd.
    238194
    239195Features that could be useful include (but are not limited to):
    240 * Not using daemondo if the daemon works fine under launchd without it
    241 * Ability to install multiple plists
    242 * Support for LaunchAgents as well as LaunchDaemons
    243 * Installing plists in ~/Library for non-root installs if the user wants
    244 * only modify specific XML tags to avoid clobbering additions by user
     196 - Not using daemondo if the daemon works fine under launchd without it
     197 - Ability to install multiple plists
     198 - Support for LaunchAgents as well as LaunchDaemons
     199 - Installing plists in ~/Library for non-root installs if the user wants
     200 - only modify specific XML tags to avoid clobbering additions by user
    245201
    246202* Difficulty: Easy
     
    250206==== Reclaim disk space ==== #reclaim
    251207
    252 Improve `port clean` to be able to delete distfiles for a specified
    253 version, and all distfiles not needed by currently installed versions of
    254 ports. In general, add an action for reclaiming disk space, which would
    255 delete old distfiles and archives and uninstall inactive ports (and
    256 anything else that would help).
     208Improve `port clean` to be able to delete distfiles for a specified version, and all distfiles not needed by currently installed versions of ports. In general, add an action for reclaiming disk space, which would delete old distfiles and archives and uninstall inactive ports (and anything else that would help). Implement heuristics to detect common hogs of disk space such as lots of old distfiles and inactive ports and periodically suggest running a cleanup command to users if such possibly unneeded files are detected.
    257209
    258210* Difficulty: Easy
     
    262214==== Parallel execution ==== #parallel
    263215
    264 When an action will run targets on multiple ports, run them in parallel
    265 when possible and sensible (requires tracking dependencies between both
    266 targets and ports and figuring out the maximum reasonable parallelism,
    267 e.g. several ports can fetch at once on a fast connection but you only
    268 want one 'make -j8' at a time).
    269 
    270 * Difficulty: Very challenging
     216When an action will run targets on multiple ports, run them in parallel when possible and sensible (requires tracking dependencies between both targets and ports and figuring out the maximum reasonable parallelism, e.g. several ports can fetch at once on a fast connection but you only want one 'make -j8' at a time).
     217
     218* Difficulty: Challenging
    271219* Languages: Tcl, C
    272220* Potential mentors: TBD
     
    274222==== Migrate muniversal into base (lipo merging) ==== #muniversal
    275223
    276 Integrate the
    277 [[source:trunk/dports/_resources/port1.0/group/muniversal-1.0.tcl|muniversal portgroup]]
    278 into base. Not just a direct copy-and-paste, but in a way that makes
    279 sense and preserves the way portfiles are expected to behave (which the
    280 current portgroup doesn't).
     224Integrate the [[source:trunk/dports/_resources/port1.0/group/muniversal-1.0.tcl|muniversal portgroup]] into base. Not just a direct copy-and-paste, but in a way that makes sense and preserves the way portfiles are expected to behave (which the current portgroup doesn't).
    281225
    282226* Difficulty: Medium
     
    286230==== Improve fetching from version control ==== #fetchtypes
    287231
    288 Make cvs/svn/git/hg/bzr fetch types checkout into the distfiles dir and
    289 then export into the work dir, to [[ticket:16373|avoid having to re-fetch]]
     232Make cvs/svn/git/hg/bzr fetch types checkout into the distfiles dir and then export into the work dir, to [[ticket:16373|avoid having to re-fetch]]
    290233after cleaning the work directory.
    291234
     
    296239==== Interactive port command ==== #interactive
    297240
    298 Write an interactive command-line tool that can be used instead of the
    299 non-interactive port(1). (The existing "interactive mode" of port(1) is actually just batch
    300 mode reading from stdin, and is not really interactive.)
    301 Factor out code used by both tools into a shared module.
    302 
    303 An interactive tool would ask for user input to resolve many situations
    304 that cause port(1) to simply error out. For example, if you try to
    305 install a port and one of its dependencies conflicts with something
    306 already installed, it could ask if you want to deactivate the installed
    307 one and its dependents.
     241Write an interactive command-line tool that can be used instead of the non-interactive port(1). (The existing "interactive mode" of port(1) is actually just batch mode reading from stdin, and is not really interactive.) Factor out code used by both tools into a shared module.
     242
     243An interactive tool would ask for user input to resolve many situations that cause port(1) to simply error out. For example, if you try to install a port and one of its dependencies conflicts with something already installed, it could ask if you want to deactivate the installed one and its dependents.
    308244
    309245* Difficulty: Medium
     
    311247* Potential mentors: TBD
    312248
    313 
    314249=== Secondary tasks ===
    315250
    316251==== Portfiles ==== #portfiles
    317252
    318 Sweep through all Portfiles and look for useful opportunities to add
    319 more built-in Tcl functions that make Portfiles more (usefully) terse,
    320 powerful, flexible or easier to write. I'm sure there is an entirely
    321 family of helper functions yet to be written here. This might also
    322 include porting additional packages to MacPorts and cleaning up or
    323 removing obsolete ports.
     253Sweep through all Portfiles and look for useful opportunities to add more built-in Tcl functions that make Portfiles more (usefully) terse, powerful, flexible or easier to write. I'm sure there is an entirely family of helper functions yet to be written here. This might also include porting additional packages to MacPorts and cleaning up or removing obsolete ports.
    324254
    325255* Classification: Medium
     
    329259==== Documentation and website ==== #docs
    330260
    331 Improve MacPorts
    332 [query:status!=closed&component=guide|server/hosting|website|wiki documentation, website and Trac system].
    333 Note that pure documentation proposals are not allowed by Google.
     261Improve MacPorts [query:status!=closed&component=guide|server/hosting|website|wiki documentation, website and Trac system]. Note that pure documentation proposals are not allowed by Google.
    334262
    335263* Difficulty: Easy to difficult
     
    339267==== Shell environment ==== #shell-environment
    340268
    341 Add support for providing basic and port-provided environmental services
    342 to users in the `~/.profile`, `~/.cshrc`, and `~/.xinitrc` files, so
    343 that instead of manipulating the user's .profile to modify certain
    344 paths, the installer could append "`source /opt/local/etc/bash.rc`" to
    345 the end of a user's .profile file and that bash.rc would source all the
    346 files in `/opt/local/etc/bash.d`.
     269Add support for providing basic and port-provided environmental services to users in the `~/.profile`, `~/.cshrc`, and `~/.xinitrc` files, so that instead of manipulating the user's .profile to modify certain paths, the installer could append "`source /opt/local/etc/bash.rc`" to the end of a user's .profile file and that bash.rc would source all the files in `/opt/local/etc/bash.d`.
    347270
    348271This task alone is most probably not enough for the whole Summer Of Code.
     
    366289Programming languages: Tcl and C[[BR]]
    367290Potential mentor: blb
    368 }}}
    369 
    370 
    371 {{{
    372 #!comment
    373 # Removed ideas which do not fit to our plans anymore
    374 
    375 # Will be obsolete and no longer be possible with the images-and-archives branch
    376 ==== Images (Pkgview, depot-to-depot dependencies) ==== #images
    377 
    378 MacPorts implements what we call the Image mode where software are stored in a depot (e.g. `/opt/local/var/db/dports/software/gawk/3.1.5_2/opt/local/bin/gawk`). For the moment, this functions like an archive and files are hardlinked to their active location (/opt/local/bin/gawk). If port A depends on dynamic library B.dylib from port B, it actually links to /opt/local/lib/B.dylib. The idea of this task is to figure out a way for port A to be linked to the library in the depot and to work even if port B is not active. That is to say that if port foo depends on version 1.2.3 of port bar, it should be compiled and linked in such a way that it's wired to the depot location of bar, not the "activated" location.  That will finally fix the fragility problem where deactivating port bar vers n-1 in order to install port bar vers n (because other things depend on n) won't also require breaking everything that relies on n-1. This mechanism also has additional advantages such as limiting the unavailability time to a minimum, especially when upgrading libraries every other package depends on (such as gettext).
    379 
    380 Classification: challenging to very challenging task[[BR]]
    381 Programming language: Tcl[[BR]]
    382 Potential mentor: TBD
    383291}}}
    384292