TLDR: we recommend that you use pipenv on CRNCH environments rather other Python package managers or tools like conda/anaconda. See the pipenv section for more details on usage.
FAQs on CRNCH Python Environments¶
- The default Python is typically 3.8. We do not officially support Python 2 usage as most packages have updated to support Python 3.
- Look into 2to3 if your code is still using Python 2!
venvis installed across most of our servers and as many dev boards as possible.
- We recommend to use either
pipenvto install packages into your local virtual environments. Note that
pipare default packages for all Python 3.3+ installations, and
venvcontains a subset of
- We recommend to use either
- We do not typically recommend using conda, miniconda, or anaconda as these quickly eat up home directory space.
- However, if you want to use conda or miniconda please consider using miniconda with your scratch space folder to store your conda environment and venvs. See the example below under the Conda section as a template.
What’s the difference between pip, venv, env, conda, etc?¶
|Tool Name||Supported Python Versions||Purpose||Default on CRNCH RG||Notes|
|pip||All Versions||Default package manager||Y|
|pipenv||2+||Package, dependency, and environment manager||Y||Combines pip and virtualenv|
|venv||3.3+||Environment manager||Y||venv is a subset of virtualenv installed by default with Python 3.3+|
|miniconda||NA||Minimalist package and environment manager||N||Suggested version of conda to use on RG; Installs its own conda/Python as well as non-Python packages|
|anaconda||NA||Package and environment manager||N||Not supported on RG; Installs its own Python|
|poetry||3.7+||Package and dependency manager||N||Not supported on RG|
Using venv on CRNCH RG¶
Venv is the default virtual environment module included since Python 3.3, and it totally replaces pyenv since Python 3.6. Virtualenv has many similarities to venv in terms of its functionality, but we recommend using venv unless you need to use a version of Python older than 3.3.
Creating a new virtual environment with venv¶
$ mkdir myproject $ python -m venv myproject
Activating/deactivating an environment¶
$ source myproject/bin/activate //To leave type exit (myproject)gburdell@rg-login:$ exit
Installing and using packages¶
Here we demonstrate a basic usage of pip with venv. We highly recommend using pipenv, which provides a more robust combination of pip and virtual environments.
$ source myproject/bin/activate (myproject)gburdell@rg-login:$ pip install matplotlib Collecting matplotlib Downloading matplotlib-3.6.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/9.4 MB 56.5 MB/s eta 0:00:00 //Use pip freeze to generate a requirements.txt file which can be used to reinstall a specific environment in the future. pip freeze > requirements.txt (myproject)gburdell@rg-login:~/USERSCRATCH/myproject$ ls bin include lib lib64 pyvenv.cfg requirements.txt share (myproject)gburdell@rg-login:~/USERSCRATCH/myproject$ more requirements.txt contourpy==1.0.6 ... matplotlib==3.6.2 numpy==1.24.1 ... six==1.16.0
Using pipvenv on CRNCH RG¶
Pipenv combines the best parts of the pip package manager for Python and virtual environments, as typified by virtualenv and venv. One key difference is that pipenv keeps all of its dependencies for installations in a
Pipfile that can then be used to regenerate a specific environment. Pipenv uses TOML syntax, and one Pipfile can be used in place of multiple requirements.txt files created by Pip with virtual environments. The
Pipfile.lock file provides a secure hashed record of installations that can be used for future deployments.
Creating a new virtual environment with pipenv¶
When you run
pipenv install, it will create a standard virtual environnment and all related pip installs will occur within this user-accessible folder.
$ pipenv install Creating a virtualenv for this project... Pipfile: /nethome/gburdell/Pipfile Using /usr/bin/python3.8 (3.8.13) to create virtualenv... ⠦ Creating virtual environment...created virtual environment CPython3.8.13.final.0-64 in 2991ms creator CPython3Posix(dest=/nethome/gburdell/.local/share/virtualenvs/gburdell-hxKrwMjp, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/nethome/gburdell/.local/share/virtualenv) added seed packages: pip==22.3, setuptools==65.5.0, wheel==0.37.1 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator ✔ Successfully created virtual environment! Virtualenv location: /nethome/gburdell/.local/share/virtualenvs/gburdell-hxKrwMjp Pipfile.lock not found, creating... Locking [dev-packages] dependencies... Locking [packages] dependencies... Updated Pipfile.lock (db4242)! Installing dependencies from Pipfile.lock (db4242)... 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00 To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run. If you'd like to create a Python 3.8 environment, use the following syntax. Note that this will overwrite the standard location for your virtualenv
Activating/deactivating an environment¶
$ pipenv shell Launching subshell in virtual environment... . /nethome/gburdell/.local/share/virtualenvs/gburdell-hxKrwMjp/bin/activate gburdell@rg-login:~$ . /nethome/gburdell/.local/share/virtualenvs/gburdell-hxKrwMjp/bin/activate (gburdell) gburdell@rg-login:~$
OR use the code::pipenv run method
$ python3 --version Python 3.6.8 $ pipenv run python3 --version Python 3.8.13
Installing and using packages¶
$ pipenv install 2to3 Installing 2to3... Adding 2to3 to Pipfile's [packages]... ✔ Installation Succeeded Pipfile.lock (db4242) out of date, updating to (7d7dfd)... Locking [dev-packages] dependencies... Locking [packages] dependencies... Building requirements... Resolving dependencies... ✔ Success! Updated Pipfile.lock (7d7dfd)! Installing dependencies from Pipfile.lock (7d7dfd)... 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00 To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run.
To show what packages are installed and their dependencies, you can use
pipenv graph. Here we show the dependencies for 2to3 and matplotlib.
$ pipenv graph //No dependencies for this package 2to3==1.0 //Several dependencies were installed, including numpy matplotlib==3.6.2 - contourpy [required: >=1.0.1, installed: 1.0.6] - numpy [required: >=1.16, installed: 1.24.1] - cycler [required: >=0.10, installed: 0.11.0] - fonttools [required: >=4.22.0, installed: 4.38.0] - kiwisolver [required: >=1.0.1, installed: 1.4.4] - numpy [required: >=1.19, installed: 1.24.1] - packaging [required: >=20.0, installed: 23.0] - pillow [required: >=6.2.0, installed: 9.4.0] - pyparsing [required: >=2.2.1, installed: 3.0.9] - python-dateutil [required: >=2.7, installed: 2.8.2] - six [required: >=1.5, installed: 1.16.0]
Pip or the package installer for python is the default way to install packages from the Python Package Index, or PyPI. Depending on the version of Python used, you may need to call it using code::pip install <packagename> or code::python -m pip install <packagename>.
Note that best practices specify that you should install packages into a “user-local” directory (normally under
~/.local or your virtual environment folder). You can find this location for your version of Python using the following command.
$ python3 -m site --user-base /nethome/gburdell/.local
Then you can install packages to your local directory as follows. Assuming a standard Python 3.8 install, the installed files can be found at code::.local/lib/python3.8/site-packages/.
$ pip install --user matplotlib
Note: We typically don’t recommend using anaconda due to the amount of dependencies it pulls into your home directory. If you get to where you need anaconda for a project this is typically some software that should be installed in a project space or system-wide! Please consider submitting a help ticket especially if you need multiple packages that can’t be satisfied with pipenv.
Miniconda Installation and Usage Example¶
With the above caveat in mind, this example shows how to use your scratch space to install and use Miniconda. We recommend this approach since this saves space in your home directory and because full Conda environments do not typically need to be backed up. Note that you can always use
conda env export --from-history>ENV.yml to back up an installed environment.
mkdir ~/USERSCRATCH/conda gburdell@rg-login:~/tutorials$ cd ~/USERSCRATCH/conda/ gburdell@rg-login:~/USERSCRATCH/conda$ wget https://repo.anaconda.com/miniconda/Miniconda3-py38_22.11.1-1-Linux-x86_64.sh ... ... ‘Miniconda3-py38_22.11.1-1-Linux-x86_64.sh’ saved [64630241/64630241] //This command uses "batch mode" to auto-accept the EULA and installs in a local folder gburdell@rg-login:~/USERSCRATCH/conda$ bash Miniconda3-py38_22.11.1-1-Linux-x86_64.sh -b -p conda3_22.11.1 PREFIX=/nethome/gburdell/USERSCRATCH/conda/conda3_22.11.1 Unpacking payload ... Installing base environment... Downloading and Extracting Packages ... installation finished. //Add the location of miniconda to your path. You should add this to your .bashrc file export PATH=$PATH:~/USERSCRATCH/conda/conda3_22.11.1/bin && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/USERSCRATCH/conda/conda3_22.11.1/lib //Create a new conda environment on your scratch space. conda create --prefix ~/USERSCRATCH/condaenv/ Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /nethome/gburdell/USERSCRATCH/condaenv Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate /nethome/gburdell/USERSCRATCH/condaenv # # To deactivate an active environment, use # # $ conda deactivate
Poetry is a tool for dependency management and packaging similar to pipenv (which combines pip and venv). While we don’t currently support it, you may be interested to try it out in your user-local setup. Read more about Poetry at the official website.
Bonus: IPython, IPykernel, and Jupyter¶
You may see some reference to IPython kernels which switching between virtual environments or especially for Jupyter notebooks. In short, IPython (see site) is a command shell for interactive Python execution that can be extended for GUI applications and parallel computing. Jupyter is a web-based interactive tool that builds on IPython but also supports many other kernels for languages like Julia and R. You can read more about kernels for Jupyter at this link.