Python Environments

Note

TLDR: we recommend that you use pipenv on CRNCH environments rather other Python package managers or tools like conda/anaconda. See the pipenv section for more details on usage.

FAQs on CRNCH Python Environments

• The default Python is typically 3.8. We do not officially support Python 2 usage as most packages have updated to support Python 3.

  • Look into 2to3 if your code is still using Python 2!

• virtualenv and venv is installed across most of our servers and as many dev boards as possible.

  • We recommend to use either virtualenv or venv with pip or pipenv to install packages into your local virtual environments. Note that venv and pip are default packages for all Python 3.3+ installations, and venv contains a subset of virtualenv functionality.

• We do not typically recommend using conda, miniconda, or anaconda as these quickly eat up home directory space.

  • However, if you want to use conda or miniconda please consider using miniconda with your scratch space folder to store your conda environment and venvs. See the example below under the Conda section as a template.

What’s the difference between pip, venv, env, conda, etc?

Python Environment Tools

Tool Name	Supported Python Versions	Purpose	Default on CRNCH RG	Notes
pip	All Versions	Default package manager	Y
pipenv	2+	Package, dependency, and environment manager	Y	Combines pip and virtualenv
virtualenv	2+	Environment manager	Y
venv	3.3+	Environment manager	Y	venv is a subset of virtualenv installed by default with Python 3.3+
miniconda	NA	Minimalist package and environment manager	N	Suggested version of conda to use on RG; Installs its own conda/Python as well as non-Python packages
anaconda	NA	Package and environment manager	N	Not supported on RG; Installs its own Python
poetry	3.7+	Package and dependency manager	N	Not supported on RG

Using venv on CRNCH RG

Venv is the default virtual environment module included since Python 3.3, and it totally replaces pyenv since Python 3.6. Virtualenv has many similarities to venv in terms of its functionality, but we recommend using venv unless you need to use a version of Python older than 3.3.

Creating a new virtual environment with venv

$ mkdir myproject
$ python -m venv myproject

Activating/deactivating an environment

$ source myproject/bin/activate
//To leave type exit
(myproject)gburdell@rg-login:$ exit

Installing and using packages

Here we demonstrate a basic usage of pip with venv. We highly recommend using pipenv, which provides a more robust combination of pip and virtual environments.

$ source myproject/bin/activate
(myproject)gburdell@rg-login:$ pip install matplotlib
Collecting matplotlib
Downloading matplotlib-3.6.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/9.4 MB 56.5 MB/s eta 0:00:00

//Use pip freeze to generate a requirements.txt file which can be used to reinstall a specific environment in the future.
pip freeze > requirements.txt
(myproject)gburdell@rg-login:~/USERSCRATCH/myproject$ ls
bin  include  lib  lib64  pyvenv.cfg  requirements.txt  share
(myproject)gburdell@rg-login:~/USERSCRATCH/myproject$ more requirements.txt
contourpy==1.0.6
...
matplotlib==3.6.2
numpy==1.24.1
...
six==1.16.0

Using pipvenv on CRNCH RG

Pipenv combines the best parts of the pip package manager for Python and virtual environments, as typified by virtualenv and venv. One key difference is that pipenv keeps all of its dependencies for installations in a Pipfile that can then be used to regenerate a specific environment. Pipenv uses TOML syntax, and one Pipfile can be used in place of multiple requirements.txt files created by Pip with virtual environments. The Pipfile.lock file provides a secure hashed record of installations that can be used for future deployments.

Installing pipenv

Using the official installation instructions here:

python3 -m pip install pipenv

Creating a new virtual environment with pipenv

When you run pipenv install, it will create a standard virtual environnment and all related pip installs will occur within this user-accessible folder.

   $ pipenv install
   Creating a virtualenv for this project...
   Pipfile: /nethome/gburdell/Pipfile
   Using /usr/bin/python3.8 (3.8.13) to create virtualenv...
   ⠦ Creating virtual environment...created virtual environment CPython3.8.13.final.0-64 in 2991ms
     creator CPython3Posix(dest=/nethome/gburdell/.local/share/virtualenvs/gburdell-hxKrwMjp, clear=False, no_vcs_ignore=False, global=False)
     seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/nethome/gburdell/.local/share/virtualenv)
       added seed packages: pip==22.3, setuptools==65.5.0, wheel==0.37.1
       activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

   ✔ Successfully created virtual environment!
   Virtualenv location: /nethome/gburdell/.local/share/virtualenvs/gburdell-hxKrwMjp
   Pipfile.lock not found, creating...
   Locking [dev-packages] dependencies...
   Locking [packages] dependencies...
   Updated Pipfile.lock (db4242)!
   Installing dependencies from Pipfile.lock (db4242)...
     🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
   To activate this project's virtualenv, run pipenv shell.
   Alternatively, run a command inside the virtualenv with pipenv run.

If you'd like to create a Python 3.8 environment, use the following syntax. Note that this will overwrite the standard location for your virtualenv

Activating/deactivating an environment

$ pipenv shell
Launching subshell in virtual environment...
 . /nethome/gburdell/.local/share/virtualenvs/gburdell-hxKrwMjp/bin/activate
gburdell@rg-login:~$  . /nethome/gburdell/.local/share/virtualenvs/gburdell-hxKrwMjp/bin/activate
(gburdell) gburdell@rg-login:~$

OR use the code::pipenv run method

$ python3 --version
Python 3.6.8
$ pipenv run python3 --version
Python 3.8.13

Installing and using packages

$ pipenv install 2to3
Installing 2to3...
Adding 2to3 to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock (db4242) out of date, updating to (7d7dfd)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (7d7dfd)!
Installing dependencies from Pipfile.lock (7d7dfd)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

To show what packages are installed and their dependencies, you can use pipenv graph. Here we show the dependencies for 2to3 and matplotlib.

$ pipenv graph
//No dependencies for this package
2to3==1.0
//Several dependencies were installed, including numpy
matplotlib==3.6.2
- contourpy [required: >=1.0.1, installed: 1.0.6]
  - numpy [required: >=1.16, installed: 1.24.1]
- cycler [required: >=0.10, installed: 0.11.0]
- fonttools [required: >=4.22.0, installed: 4.38.0]
- kiwisolver [required: >=1.0.1, installed: 1.4.4]
- numpy [required: >=1.19, installed: 1.24.1]
- packaging [required: >=20.0, installed: 23.0]
- pillow [required: >=6.2.0, installed: 9.4.0]
- pyparsing [required: >=2.2.1, installed: 3.0.9]
- python-dateutil [required: >=2.7, installed: 2.8.2]
  - six [required: >=1.5, installed: 1.16.0]

Pipenv Related Documents

• Pipenv and Virtualenv
• Pipenv guide
• Pipenv vs virtualenv vs conda environment

Pip

Pip or the package installer for python is the default way to install packages from the Python Package Index, or PyPI. Depending on the version of Python used, you may need to call it using code::pip install <packagename> or code::python -m pip install <packagename>.

Note that best practices specify that you should install packages into a “user-local” directory (normally under ~/.local or your virtual environment folder). You can find this location for your version of Python using the following command.

$ python3 -m site --user-base
/nethome/gburdell/.local

Then you can install packages to your local directory as follows. Assuming a standard Python 3.8 install, the installed files can be found at code::.local/lib/python3.8/site-packages/.

$ pip install --user matplotlib

Pip Related Documents

• Python Pip tutorial page.

Conda

Note: We typically don’t recommend using anaconda due to the amount of dependencies it pulls into your home directory. If you get to where you need anaconda for a project this is typically some software that should be installed in a project space or system-wide! Please consider submitting a help ticket especially if you need multiple packages that can’t be satisfied with pipenv.

Miniconda Installation and Usage Example

With the above caveat in mind, this example shows how to use your scratch space to install and use Miniconda. We recommend this approach since this saves space in your home directory and because full Conda environments do not typically need to be backed up. Note that you can always use conda env export --from-history>ENV.yml to back up an installed environment.

mkdir ~/USERSCRATCH/conda
gburdell@rg-login:~/tutorials$ cd ~/USERSCRATCH/conda/
gburdell@rg-login:~/USERSCRATCH/conda$ wget https://repo.anaconda.com/miniconda/Miniconda3-py38_22.11.1-1-Linux-x86_64.sh
...
... ‘Miniconda3-py38_22.11.1-1-Linux-x86_64.sh’ saved [64630241/64630241]
//This command uses "batch mode" to auto-accept the EULA and installs in a local folder
gburdell@rg-login:~/USERSCRATCH/conda$ bash Miniconda3-py38_22.11.1-1-Linux-x86_64.sh -b -p conda3_22.11.1
PREFIX=/nethome/gburdell/USERSCRATCH/conda/conda3_22.11.1
Unpacking payload ...
Installing base environment...
Downloading and Extracting Packages
...
installation finished.

//Add the location of miniconda to your path. You should add this to your .bashrc file
export PATH=$PATH:~/USERSCRATCH/conda/conda3_22.11.1/bin && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/USERSCRATCH/conda/conda3_22.11.1/lib

//Create a new conda environment on your scratch space.
conda create --prefix ~/USERSCRATCH/condaenv/
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##
environment location: /nethome/gburdell/USERSCRATCH/condaenv

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate /nethome/gburdell/USERSCRATCH/condaenv
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Conda Related Documents

• Understanding conda and pip
• Explaining the many flavors of conda
• OLCF’s guide to using conda, which assumes a sitewide installation of conda.
• NERSC’s guide on using python with anaconda

Poetry

Poetry is a tool for dependency management and packaging similar to pipenv (which combines pip and venv). While we don’t currently support it, you may be interested to try it out in your user-local setup. Read more about Poetry at the official website.

Bonus: IPython, IPykernel, and Jupyter

You may see some reference to IPython kernels which switching between virtual environments or especially for Jupyter notebooks. In short, IPython (see site) is a command shell for interactive Python execution that can be extended for GUI applications and parallel computing. Jupyter is a web-based interactive tool that builds on IPython but also supports many other kernels for languages like Julia and R. You can read more about kernels for Jupyter at this link.