7 votes

How do you keep pypi programs up to date?

Posted by annadane
Tags: python, packaging, pip, virtualenvs, ask.help

If you install a bunch of stuff from pypi (like in virtualenv), how do you keep abreast of security fixes and updates etc?

9 comments

  1. Deimos
    Link
    I think it depends exactly what you're doing, but for larger projects I usually try to use a method similar to this one: A Better Pip Workflow. That's what I do for Tildes - there are two separate...

    I think it depends exactly what you're doing, but for larger projects I usually try to use a method similar to this one: A Better Pip Workflow.

    That's what I do for Tildes - there are two separate files related to the pypi packages I'm using:

    • requirements-to-freeze.txt - these are the "real" packages that I want to install. Just a list of the ones I want, preferably with no particular version requirements unless it's necessary to restrict a version for some reason.
    • requirements.txt - this is what actually gets installed if I install the requirements-to-freeze list, including all dependencies and specific versions of each package. This file basically lists a complete "snapshot" of the packages in the virtualenv.

    Now when I want to update the pypi packages, I go through this process:

    1. Delete the project's virtualenv completely
    2. Recreate it, and then do pip install -r requirements-to-freeze.txt. This installs the newest versions of all the packages and their dependencies.
    3. Run pip freeze > requirements.txt. This overwrites the previous "full installed snapshot" with the new one.
    4. Use git diff to see which packages have been upgraded to new versions, and potentially visit their docs/repos/etc. to check their changelogs if I want to know what changed.
    5. Run tests/etc. to try to make sure that everything's still working.
    6. Commit the changes to requirements.txt (example of what these commits look like)

    Kenneth Reitz (who wrote the workflow post this is based on) has also been working on Pipenv, which is supposed to include some of this natively and improve the process, so that may be an option as well if you're open to trying something new.

    3 votes
  2. [6]
    Lynx
    Link
    for pip, there are a few suggestions here: https://stackoverflow.com/questions/2720014/upgrading-all-packages-with-pip I just use pacman and AUR packages though, one package manager on a system is...

    for pip, there are a few suggestions here: https://stackoverflow.com/questions/2720014/upgrading-all-packages-with-pip

    I just use pacman and AUR packages though, one package manager on a system is plenty.

    2 votes
    1. [5]
      arghdos
      Link Parent
      Eek. System python based packages are almost always far too limited/old for my use cases. This is probably the answer if you only use python casually, or with fairly standard packages but won't...

      I just use pacman and AUR packages though, one package manager on a system is plenty.

      Eek. System python based packages are almost always far too limited/old for my use cases. This is probably the answer if you only use python casually, or with fairly standard packages but won't help you at all if you get even a bit off the beaten path.

      I would recommend Conda (particularly miniconda) as the best python package manager currently. Something like a hybrid between pip & virtualenv it can manage separate environments without a sweat, and upgrading / installing packages is simple (even on Windows, where pip usually sucks to try to set up correctly) as they're distributed in pre-built binaries (conda update --all)

      3 votes
      1. [4]
        Lynx
        Link Parent
        Personally I've only ever found one python package that wasn't in the AUR, so I added it. Maintainers are usually pretty good at keeping up to date with releases too. But sure, if you need...

        System python based packages are almost always far too limited/old for my use cases.

        Personally I've only ever found one python package that wasn't in the AUR, so I added it. Maintainers are usually pretty good at keeping up to date with releases too. But sure, if you need something actually bleeding-edge (i.e. git master), or need to develop your own python software, use a virtualenv, possibly with a wrapper like Conda.

        2 votes
        1. [3]
          arghdos
          Link Parent
          Is AUR more up-to-date / extensive than typical package repos on apt or yum? It sure sounds it :)

          Is AUR more up-to-date / extensive than typical package repos on apt or yum? It sure sounds it :)

          1. Lynx
            Link Parent
            The AUR are just user produced build files, and they usually get updated (or flagged out-of-date) as soon as a new upstream version is out. Arch in general is rolling-release, so most software in...

            The AUR are just user produced build files, and they usually get updated (or flagged out-of-date) as soon as a new upstream version is out. Arch in general is rolling-release, so most software in the repositories is at most a week or two out of date. Debian tends to be more on the scale of months or years.

            5 votes
          2. fifthecho
            Link Parent
            AUR is the Arch User Repository. It's a wild-west of user contributed package build scripts. Sometimes quality can be so-so (though, really, there's only been a few instances of people getting...

            AUR is the Arch User Repository. It's a wild-west of user contributed package build scripts.

            Sometimes quality can be so-so (though, really, there's only been a few instances of people getting malicious PKGBUILD scripts into the AUR) but as it's community-run, it tends to be pretty close to the bleeding edge.

            I recall when Microsoft first released VS Code (IIRC only for Ubuntu) and there was a PKGBUILD within 2 hours that unpacked the Debian package, moved things to where Arch would expect them, and then bundle it up.

            Additionally, as Arch is built around being a rolling-release distribution, it should ALWAYS be at (or very closely following) the latest stable release of most any piece of software. Debian, Ubuntu, Centos, RHEL, and Fedora all stamp a "stable" release and then don't really push new versions of software. Arch is never "stable" (though the platform itself tends to be more often than not) so because there isn't an arbitrary line drawn in the sand, they can push updates like this.

            Arch is great for a desktop/laptop machine...but unless you're containerizing all the things, I wouldn't use it for a server because you never know when something might break (or an update to a library might break your software).

            4 votes
  3. teaearlgraycold
    Link
    If you use Anaconda it has a command for updating all of your packages.

    If you use Anaconda it has a command for updating all of your packages.

    2 votes
  4. joelthelion
    Link
    I try to use my distro's package manager instead of pip when possible. In general I try to avoid having too many dependencies. But it's a hard problem anyways.

    I try to use my distro's package manager instead of pip when possible. In general I try to avoid having too many dependencies. But it's a hard problem anyways.