Skip to content
Theory · NumPy and pandas · lesson 10 of 12

Grouping and the median

The average salary over a whole table says little if the table holds several teams. You need the mean for each team separately. In pandas this takes no loop:

Python
import pandas as pd

df = pd.DataFrame({
    "team": ["ml", "ml", "ml", "web", "web", "web", "web"],
    "level": ["jun", "mid", "sen", "jun", "jun", "mid", "sen"],
    "salary": [100, 120, 140, 90, 95, 105, 610],
})
means = df.groupby("team")["salary"].mean()
print(means.to_dict())  # {'ml': 120.0, 'web': 225.0}
print(df.groupby("team")["salary"].count().to_dict())  # {'ml': 3, 'web': 4}

Grouping

Grouping splits rows by the value of a column. The call df.groupby("team") collects the rows with "ml" into one group and the rows with "web" into another. Then you select a column, ["salary"], and compute one number per group: mean, median, sum or count.

The result is a Series column where each group has its number. The to_dict method turns it into a dictionary. The count method counts filled values, so web has four. Missing values are skipped: if a group holds only missing values, mean and median give it NaN, and count gives 0.

Median

The web team has a mean of 225.0, although three of its four people earn 105 or less. One salary of 610 pulls the mean up. The median is the value in the middle of the sorted list, and one large number barely affects it.

To find the median, the values are sorted: for web they are 90, 95, 105, 610. For an even count, the median is the mean of the two central values: for web, (95 + 105) / 2.

Python
medians = df.groupby("team")["salary"].median()
print(medians.to_dict())  # {'ml': 120.0, 'web': 100.0}

Salaries, prices and incomes usually form skewed data: most values are small, and rare ones are very large. On such data the median describes a typical object better than the mean. In the ml team the values are symmetric, and both measures give 120.0.

If the mean is well above the median, the data has rare large values. The mean is still useful: multiplied by the headcount, it gives the payroll, and the median does not.

Common mistake

The mean is computed over the whole grouping without selecting a column:

Python
print(df.groupby("team").mean())  # ✗
TEXT
TypeError: agg function failed [how->mean,dtype->object]

The table has a text column level, and pandas tries to average it too. Before version 2.0, pandas dropped such columns, so old examples often contain this call. Select a numeric column before computing: df.groupby("team")["salary"].mean().

Subscribe to take the course

The course is free. Sign in with Telegram and subscribe to the channel to open every lesson.

Sign in another way