I recently got around to hacking a system for managing my
conda environments better. Previously, my coding projects mostly relied on one master environment (with exceptions, e.g.
bokeh development, or my Network Analysis Made Simple tutorial), but conflicts started cropping up. Thus, I decided to separate out my environments. However, keeping track of which environments go with which projects began getting tedious.
I thus decided to automate some of the steps involved in maintaining environments, and keep everything centrally managed so my brain doesn't overload. It involves a bit of GitHub and a bit of bash scripting, but altogether gives a ton of flexibility and control over keeping my environments updated.
I start by keeping a central repository of
conda environment YAML specifications. Mine is kept here. Each YAML specification includes just the minimum set of packages that I need;
conda manages the dependencies.
For example, my environment specification for Bayesian statistical analyses looks as such:
name: bayesian # for Bayesian analysis channels: !!python/tuple - conda-forge - defaults - ericmjl dependencies: - python=3.6 - matplotlib - numpy - pandas - scipy - seaborn - pymc3 - jupyter - jupyterlab
Now, I've not pinned specific versions here, because I like to keep up with the latest stable releases. However, if version pinning is desired, it's totally possible to pin specific packages to particular versions, using the same syntax as I did for
In each project repository, I have an
update_env.sh script, that looks something like this:
wget https://raw.githubusercontent.com/ericmjl/conda-envs/master/lektor.yml -O environment.yml conda env update -f environment.yml
The key idea here is that I download only the relevant YAML file, export it as a generic
environment.yml file, and then run the
conda env update command on it to keep the environment up-to-date.
Now, here's the magic. I hacked Christine Doig's
conda-auto-env script to execute
update_env.sh, and then auto-activate the environment.
If my environment needs change, I can always update the environment YAML spec file (e.g.
bayesian.yml) in the central repository, and use that to automatically update individual project environments.