The pandas DataFrame

In a numpy array, all elements share one type, and the columns have no names. Real data looks different: one column holds a city, another an age, and columns are easier to reach by name. The pandas library exists for such data:

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "city": ["Kazan", "Omsk", "Kazan", None],
    "age": [25, 41, np.nan, 33],
    "salary": [90, 120, 75, 110],
})
print(df.shape)  # (4, 3)
print(df["age"].mean())  # 33.0
print(df[["city", "age"]].shape)  # (4, 2)

A DataFrame table is a set of named columns, and each column has its own type. It is convenient to create one from a dictionary: the key is the column name, the value is a list of values. The shape attribute works as for an array: four rows, three columns.

Columns and rows

A name in square brackets returns one column—a Series. It has the same methods as an array: mean, sum, comparison with a number. A list of names, df[["city", "age"]], returns a new table of those columns.

Comparing a column with a number gives a boolean mask, and a mask in square brackets keeps the matching rows. The columns of one table are aligned: df["age"] and df["salary"] describe the same people in the same order. So a mask built from one column selects rows of the whole table:

older = df[df["age"] > 30]
print(older["salary"].tolist())  # [120, 110]

The tolist method turns a column into a plain Python list.

Missing values

A missing value is a cell without a value. In a numeric column pandas stores it as NaN, and a text column may hold None. The isna method marks missing values of both kinds with True, and notna does the opposite, marking filled cells.

print(df.isna().sum().tolist())  # [1, 1, 0]
known = df[df["age"].notna()]
print(known.shape)  # (3, 3)

On a table, sum adds down each column by default. So df.isna().sum() gives the number of missing values in each column: one each in city and age. The mean 33.0 in the first example skips the missing value: mean, sum and other pandas computations ignore them. Missing values are counted before a model is trained: most scikit-learn models stop with a ValueError on NaN.

Common mistake

Missing values are searched for by comparing with np.nan:

missing = df["age"] == np.nan  # ✗
print(missing.sum())  # 0

Python reports no error, but zero missing values are found. NaN equals nothing, not even itself, so the comparison gives False in every row. Only df["age"].isna() finds the missing values.