← return to practice.dsc10.com
Instructor(s): Janine Tiefenbruck
This exam was administered online. Students were allowed one page of double-sided handwritten notes. No calculators were allowed. Students had 60 minutes to take this exam.
Note (groupby / pandas 2.0): Pandas 2.0+ no longer
silently drops columns that can’t be aggregated after a
groupby, so code written for older pandas may behave
differently or raise errors. In these practice materials we use
.get() to select the column(s) we want after
.groupby(...).mean() (or other aggregations) so that our
solutions run on current pandas. On real exams you will not be penalized
for omitting .get() when the old behavior would have
produced the same answer.
You have a DataFrame, books, where the rows represent
individual books and the columns are "title",
"author", "genre", and "price" as
a float in dollars.
Suppose both of the following DataFrames have the same number of rows.
books.groupby(["author", "genre"]).count()
books.groupby("author").count()In one sentence, interpret what this means about authors and genres.
In the dataset, there is no author with books in multiple genres. Equivalently, each author writes in only one genre.
When you call books.groupby(“author”).count(), pandas makes one row per unique author. When you call books.groupby([“author”, “genre”]).count(), it makes one row per unique (author, genre) pair. So, if an author writes in only one genre, then that author corresponds to exactly one (author, genre) pair. If an author writes in multiple genres, then that same author will correspond to multiple (author, genre) pairs, so the second groupby will have more rows than the first.
So if the two groupby results have the same number of rows, there cannot be any author with books in more than one genre.
Write one line of code that creates a DataFrame with
the same contents as books, except omit books where the
mean price of all books in that genre is less than 15 dollars.
books.groupby("genre").filter(lambda df: df["price"].mean() >= 15)In this solution, first, pandas splits the table of books into smaller groups based on their genre, so all science fiction books are in one group, all mystery books in another, and so on. Next, it calculates the average book price inside each genre group individually. Finally, it checks that calculated average against 15 dollars. If a genre’s average price is 15 dollars or higher, every single book from that genre stays in your final table. If a genre’s average is below 15 dollars, every book belonging to that genre gets completely filtered out.
Write one line of code that evaluates to a Series
indexed by "genre", containing a count of the number of
books in that genre that are priced above the average price for that
genre.
books.groupby("genre")["price"].agg(lambda s: (s > s.mean()).sum())The solution works by isolating each genre’s prices, calculating that specific genre’s average, and counting how many books beat that benchmark. First, groupby(“genre”)[“price”] separates all the books into groups based on their genre and isolates just the price column for each group. Next, .agg() uses a lambda calculation to compare each price to the mean of each genre group individually.
Suppose DataFrames a and b do not have any
missing values. We merge them as follows:
merged = a.merge(b, on="key", how="left")In one sentence, justify whether it is possible for
merged to have any missing values.
It is possible, because there may be a row in a for
which there is no match in b, in which case that row will
appear in the output but the columns from b will have null
values.
Suppose we merge DataFrames c and d as
follows:
result = c.merge(d, on="key", how="right")You are told that result and d have the
same number of rows. Which of the following statements are
necessarily true? Select all that
apply.
c and d have the same number of rows.
Every key that appears in c also appears in
d.
Every key that appears in d also appears in
c.
No key that appears in c appears more than once in
d.
No key that appears in d appears more than once in
c.
All the keys in c are distinct (no duplication).
All the keys in d are distinct (no duplication).
Answer: Statement 5.
Statement 5 is necessarily true: If any key in d matched multiple rows in c, the resulting DataFrame would contain more rows than d, violating the given condition.
Statements 1, 2, 3, 4, 6, and 7 are not necessarily true:
1, 6: DataFrame c can contain extra rows or duplicate keys that never match d, and these rows will simply be dropped during the right join without affecting the final row count.
2, 3: A right join retains all rows from d even if their keys do not exist in c (resulting in NaN values), meaning keys in d do not have to appear in c, and vice versa.
4, 7: DataFrame d can contain duplicate keys as long as each of those duplicate rows matches exactly one row in c
Which of these questions require a permutation test? Select all that apply.
Do more people own their home or rent their home?
Do one quarter of people have blue eyes and three quarters of people have eyes that are not blue?
Are pink clothes more expensive than clothes that are not pink?
Answer: Statement 3 only.
Statement 3 compares a numerical statistic (price) between two independent groups (pink vs. not pink). Permutation tests shuffle group labels to evaluate if observed differences are significant. Statement 1 evaluates a single categorical proportion (own vs. rent), and Statement 2 also tests a specific hypothesized proportion against observed data. These do not necessitate a permutation test.
You want to test whether left-handed and right-handed people have the same distribution of intelligence quotient (IQ) or whether right-handed people have lower IQs, on average.
You plan to simulate values of a test statistic (in the array
stats), calculate an observed value of the statistic
(obs), and compute the p-value using
(stats >= obs).mean(). Which test statistic(s) could you
use? Select all that apply.
Difference in mean IQs (right minus left).
Difference in mean IQs (left minus right).
Absolute difference in mean IQs.
Total variation distance.
Kolmogorov-Smirnov statistic.
Answer: 2 only.
We are testing if right-handed people have lower IQs on average
(\mu_{right} < \mu_{left}), which is
equivalent to testing (\mu_{left} -
\mu_{right} > 0) The P-value code
(stats >= obs).mean() checks how many simulated values
are greater than or equal to the observed value, meaning our test
statistic must yield a larger value when the alternative hypothesis is
true.
Why Statement 2 works: Using left minus right makes obs positive when left-handed IQs are higher. More extreme support for the alternative results in even larger positive numbers, which correctly aligns with stats >= obs.
Statement 1 fails as using right minus left makes obs negative under the alternative. Statement 3 fails as absolute difference measures distance from zero regardless of direction, making it suitable only for two-sided tests. Statement 4 and 5 fail as Total variation distance and the Kolmogorov-Smirnov statistic are also designed for two-sided tests.
In the DataFrame cookbook, each row represents a recipe,
and the "ingredients" columns records the number of
ingredients in the recipe. The "category" column contains
either "savory" or "sweet". Define
means as follows:
means = cookbook.groupby("category")["ingredients"].mean()Which of the following computes the average number of
"ingredients" for "savory" recipes minus the
average number of "ingredients" for "sweet"
recipes? Select all that apply.
means.diff().iloc[-1]
means.diff().iloc[-1] * -1
means.diff().sum() * -1
means.diff().iloc[0]
means.iloc[0] - means.iloc[1]
means.iloc[1] - means.iloc[0]
Answer: 2, 3, 5.
Grouping by “category” (“savory” and “sweet”) creates a Series with two entries. Alphabetical sorting determines their order: [“savory”, “sweet”]. Therefore, means.iloc[0] is savory and means.iloc[1] is sweet.
Statement 5 (means.iloc[0] - means.iloc[1]): directly
computes savory minus sweet. Statement 2 targets the last element
(“sweet”), which equals sweet - savory. Multiplying by -1 flips it to
savory minus sweet. Statement 3 produces [NaN, sweet - savory]. The
.sum() collapses this to sweet - savory. Multiplying by -1 yields savory
minus sweet.
Why the others fail: Statement 1 evaluates to sweet - savory instead of savory minus sweet. Statement 4 evaluates to NaN. Statement 6 evaluates to sweet - savory.
In the DataFrame fitness, the "steps"
column records how many steps Noah took each day, and the
"day" column records the day of the week
(e.g. "Monday"). Some values in "steps" are
missing at random (MAR) dependent on "day".
Fill in the blanks to perform mean imputation conditional on
day of the week. The resulting DataFrame, filled,
should have no missing values in "steps".
def cond_mean_impute(steps):
steps = steps.copy()
steps[steps.isna()] = ___(i)___
return steps
filled = fitness.copy()
filled["steps"] = (fitness.___(ii)___.___(iii)___)steps.mean(): Taking .mean() calculates the average
steps for that specific day.
groupby("day")["steps"] groups the DataFrame rows by
the “day” column so that missing values can be filled using statistics
calculated independently for each day.
transform(cond_mean_impute): The .transform() method
applies the imputation function to each group and returns a Series with
the exact same shape as the original column, replacing each missing
value with its respective day’s mean.
We now want to perform probabilistic imputation conditional
on day of the week. Fill in blank (i) above in a
different way, such that keeping the rest of the code unchanged
correctly accomplishes this.
np.random.choice(steps.dropna(), steps.isna().sum())np.random.choice(steps.dropna(), steps.isna().sum())
randomly samples from the observed step counts of that specific day to
fill in the missing values. This approach passes each day’s group of
data into the function, where steps.dropna() isolates the
non-missing values for that day and steps.isna().sum()
determines how many replacements are needed.
Suppose that Noah’s actual daily step counts,
including those that were missing from fitness, form a bell
curve centered at 10,000 with a standard deviation of 2,000.
Part (b)’s strategy is better. Probabilistic imputation better represents the true distribution as mean imputation collapses variance by replacing missing values with a single constant (the mean), artificially spiking the center of the distribution.
Part (a)’s strategy will give a standard deviation less than 2,000 (due to reduced variance). Part (b)’s strategy will give a standard deviation approximately equal to 2,000 because it preserves the underlying variability by drawing from the observed distribution.
At a university, an end-of-term course survey asks students for their
"instructor rating", "grade", and
"graduation year". This table shows the distribution of
"grade" for survey responses where the
"instructor rating" was missing, and the corresponding
distribution for where it was not missing (observed). Note that both
columns sum to 1.
We want to do a permutation test to determine whether the missingness
of "instructor rating" might be dependent on
"grade".
Which test statistic should be used for this permutation test? Calculate the observed value of this statistic, showing your work.
Total variation distance (TVD). 0.18
\text{TVD} = \frac{1}{2} \sum_{i} \vert{}p_i - q_i\vert{}
Summing the absolute differences:
\sum \vert{}p_i - q_i\vert{} = 0.14 + 0.01 + 0.03 + 0.03 + 0.15 = 0.36
Multiplying by \frac{1}{2}:
\text{TVD} = \frac{0.36}{2} = 0.18.
Suppose the p-value is 0.71 and we have previously determined that
"instructor rating" is neither missing by design (MD) nor
missing not at random (MNAR). Can we conclude that
"instructor rating" is missing completely at random (MCAR)?
Justify your answer.
No. Failing to reject the null only shows that there is no detected
dependence between missingness and "grade" specifically. It
says nothing about dependence on other observed variables or on the
rating value itself. Concluding MCAR would require independence from all
of these.
Justify in one sentence why the
"instructor rating" could actually be missing not at random
(MNAR).
Perhaps students who don’t like the professor don’t want to say so, so they just leave the question blank. Or perhaps students with moderate opinions leave the question blank.
The DataFrame grades is indexed by unique
"PID" numbers. It has one student "name"
column and many columns for student scores on various course
assignments. Missing values (NaN) occur if and only if a
student did not submit a particular assignment.
For each part, write one line of code to answer the
question. Use numpy and pandas operations, and
avoid loops.
Create a Series indexed by "PID" with the number of
assignments each student submitted.
grades.drop(columns=["name"]).notna().sum(axis=1)Also can use .drop("name", axis=1) which is equivalent
to .drop(columns=["name"]).
Also can use .notnull() which is equivalent to
.notna().
Also can use .count(axis=1) which drops NaNs
automatically, instead of .notna().sum(axis=1)
Another solution is to not drop the name column, then subtract 1 from the Series to account for the fact that names will be non-null.
Without sorting, find the name of the student with
the highest score on "Project 2", assuming there were no
ties.
grades.set_index("name")["Project 2"].idxmax()or
grades.loc[grades["Project 2"].idxmax(), "name"]Without grouping, find the most common score on
"Homework 4", if we consider students who did not submit
the assignment as earning a score of 0.
grades["Homework 4"].fillna(0).value_counts().index[0]Can also use .idxmax() instead of
.index[0].