Virtual Environments, Packages, Import/Export

Enrico Toffalini

Virtual Environments

→ isolated Python environments that allow you to install packages and dependencies separately from your system/main Python installation. Helps avoid conflicts across projects that may require different versions of the same packages.

→ routinely used in professional projects, they are best practice for managing projects ensuring computational reproducibility

btw something similar exists in R via the renv package, although unfortunately not widely used

Virtual Environments

Practically, they are local folders with an isolated Python environment. They contain:

  • a full copy of the Python interpreter / a symlink to the main Python installation;
  • all the packages you install at the exact versions used at that time

This folder is ideally placed inside your project directory

Virtual Environments

Create a virtual environment with this command in your bash/terminal:

python -m venv <nameOfVenv>

then, just before using, activate it:

source <nameOfVenv>/bin/activate     # Linux/macOS

<nameOfVenv>\Scripts\activate        # Windows cmd / powershell
    # or alternatively
<nameOfVenv>\Scripts\activate.ps1    # Windows powershell

… from inside an IDE, you may activate the venv via specific commands like reticulate::use_virtualenv("nameOfMyVenv", required=T) (in R / RStudio), or setting the Python interpreter manually and then restarting the kernel (in Spyder)

Virtual Environments

  • In your newly created venv, you (re)install all packages required by your project.
  • Then, at any time, you can export a requirements.txt file to document the exact versions of all installed packages (this is particularly useful for sharing your environment, e.g., via GitHub).
  • This is best practice to isolate projects and ensure reproducibility.
pip freeze > requirements.txt

Virtual Environments

  • Finally, if someone shares their project with you, or you re-install it on a different computer, you can simply create a new venv, activate it, and install all required packages at the exact versions specified in requirements.txt
pip install -r requirements.txt

# alternatively 

python -m pip install -r requirements.txt

Project like a pro 😎

  • Use a different virtual environment per project (as a subfolder);
  • Place scripts, notebooks, and data in different subfolders;
  • Use relative paths (e.g., "data/myfile.csv");
  • Save outputs, tables, figures… via code (not manually)
myProjectFolder/
├── venv/             ← virtual environment
├── data/             ← .csv, .xlsx, etc.
├── scripts/          ← .py (also .R) scripts
├── outputs/          ← different types of output files
├── notebooks/        ← markdowns, colab notebooks, etc.
├── paper/            ← in case you are writing a paper...
├── tables/           ← possibly convenient to store paper's tables
├── figures/          ← possibly convenient to store paper's figures
├── requirements.txt  ← list of installed packages for reproducibility
└── README.md         ← brief description of the project

Installing and importing packages

Installing, inside an IDE console or Colab:

!pip install numpy  # install one package
!pip install numpy pandas seaborn  # install multiple packages
!pip install numpy==2.2.5  # install package and specify which version 
!pip install -r requirements.txt  # install packages listed in requirements.txt

Then, before using any of their functions, import packages and modules:

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

“as” gives a shorter alias to a package or module name (e.g., pd for pandas; np for numpy); this is convenient because in Python you frequently need to call different functions by always specifying the package/module name (unlike in R; unless you import individual functions, e.g., from numpy import array)

Using functions, help, autocomplete

Use a function from a package, and call help:

np.mean([2.0, 1.5, 7.1, 4.2]) # call a function from a package
?np.mean      # help only in IDE console or Colab,         
help(np.mean) # help via built-in function

Use tab to autocomplete and explore available functions of a package ↴

Using functions, help, autocomplete

As in R, you can rely on positional order of arguments instead of naming them, or you can completely omit them if there are valid default arguments. However, it’s best practice to make all relevant arguments explicit for readability and reproducibility

sns.regplot(data=myDF, x="Age", y="Score", scatter=True, ci=95)
plt.show()

Accessing functions as methods

In Python, objects may have functions attached to them: these are called methods, and are accessed using dot (“.”) notation (more on this later!)

import numpy as np
myVect = np.array([2.0, 1.5, 7.1, 4.2])
Method-style syntax, requires that the object has this method: 
myVect.mean()
np.float64(3.7)
Function-style syntax
(more like R): 
np.mean(myVect)
np.float64(3.7)

Use tab to autocomplete and explore available methods of an object →

Working directory

Equivalent to R’s getwd() / setwd(): 
import os

os.getcwd()               # Get current working directory
os.chdir("path/to/folder") # Change working directory

(in Colab, paths are relative to the notebook location in Google Drive)

Relative paths

path = "myProjectFolder/"
path = "../myProjectFolder/"

Absolute paths

path = "C:/Users/enric/Desktop/myProjectFolder/" # windows-style
path = "/home/enric/Desktop/myProjectFolder/" # linux-style
path = "/Users/enric/Desktop/myProjectFolder/" # macOS-style

Import/Export objects (no equivalent of save.image() of R)

myObject = [10.5, 2, 63.11]

import pickle

with open("myObject.pkl", "wb") as file:       # "wb" is for "writing"
    pickle.dump(myObject, file)                # Save an object 
    
with open("myObject.pkl", "rb") as file:       # "rb" is for "reading"
    myObject = pickle.load(file)               # Load & assign object

compact version

pickle.dump(myObject, open("myObject.pkl", "wb")) # Save an object 

myObject = pickle.load(open("myObject.pkl", "rb")) # Load & assign object

(the compact version is suboptimal because it doesn’t properly close the file after using, but still works)

Import tabular data (more on pandas later!)

from CSV

import pandas as pd

df = pd.read_csv("data/Courses40Cycle.csv")

from Excel

df = pd.read_excel("data/Courses40Cycle.xlsx")

from Ctrl+C copied elements (beautiful ❤️ but only for Windows)

df = pd.read_clipboard()


Export tabular data

df.to_csv("exported_data.csv", index=False) # index=False excludes row numbers

df.to_excel("exported_data.xlsx", index=False)

Export figures

import matplotlib.pyplot as plt
plt.scatter(np.random.normal(0,1,10), np.random.normal(0,1,10))
plt.title("Example Plot")
plt.show()

plt.savefig("myPlot.png", dpi=300)  # export with desired resolution


Delete an object (similar to “rm(df)” in R)

del df


Listing all objects in workspace (similar to “ls()” in R)

dir()
['Age', 'N', 'Score', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'df', 'myVect', 'np', 'pd', 'pl', 'plt', 'r', 'sns']

dir()

dir() is a built-in function that does more than just returning a list of objects in workspace; it allows you to inspect all attributes and methods of any object

a = [5.2, 2, 0, 98, 11.5, 63.11]
dir(a)[37:47]
['append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse']
a = np.array([5.2, 2, 0, 98, 11.5, 63.11])
dir(a)[97:127]
['all', 'any', 'argmax', 'argmin', 'argpartition', 'argsort', 'astype', 'base', 'byteswap', 'choose', 'clip', 'compress', 'conj', 'conjugate', 'copy', 'ctypes', 'cumprod', 'cumsum', 'data', 'device', 'diagonal', 'dot', 'dtype', 'dump', 'dumps', 'fill', 'flags', 'flat', 'flatten', 'getfield']
import numpy as np
dir(np)[40:70]
['abs', 'absolute', 'acos', 'acosh', 'add', 'all', 'allclose', 'amax', 'amin', 'angle', 'any', 'append', 'apply_along_axis', 'apply_over_axes', 'arange', 'arccos', 'arccosh', 'arcsin', 'arcsinh', 'arctan', 'arctan2', 'arctanh', 'argmax', 'argmin', 'argpartition', 'argsort', 'argwhere', 'around', 'array', 'array2string']