neurostatx.utils.preprocessing

Functions

Name	Description
binary_to_yes_no	Recode 1/0 columns to Yes / No in place.
compute_correlation_coefficient	Compute a Pearson correlation matrix and write a heatmap.
compute_pca	Fit a PCA and return scores, diagnostics, and the model.
compute_shapiro_wilk_test	Run a Shapiro-Wilk normality test on each column.
get_column_indices	Return integer positions for the requested column names.
merge_dataframes	Join several tables on a shared index column.
plot_distributions	Write a histogram and ECDF for each column in df.
remove_nans	Split a table into rows with missing values and complete rows.
rename_columns	Rename selected columns and return a copy of the table.

binary_to_yes_nofunction#

View source

def binary_to_yes_no(df, cols)

Recode 1/0 columns to Yes / No in place.

Values of 1 become Yes, 0 become No, and other values become Don't know or missing value.

Parameters

Name	Type	Description
df	pd.DataFrame	Input table. Matching columns are modified in place.
cols	list of str	Column names to recode.

Returns

dfpd.DataFrame

The same table after recoding.

Examples

>>> import pandas as pd
>>> from neurostatx.utils.preprocessing import binary_to_yes_no
>>> df = pd.DataFrame({"flag": [1.0, 0.0]})
>>> binary_to_yes_no(df, ["flag"])["flag"].tolist()
['Yes', 'No']

compute_correlation_coefficientfunction#

View source

def compute_correlation_coefficient(df, out_folder, context='poster', font_scale=0.2, cmap=None, annot=False)

Compute a Pearson correlation matrix and write a heatmap.

Parameters

Name	Type	Default	Description
df	pd.DataFrame	required	Input table.
out_folder	str	required	Directory where correlation_heatmap.png is written.
context	str	'poster'	Seaborn context. Defaults to "poster".
font_scale	float	0.2	Seaborn font scale. Defaults to 0.2.
cmap	str	None	Heatmap colormap. Defaults to None.
annot	bool	False	If True, write correlation values on the heatmap. Defaults to False.

Returns

corr_matpd.DataFrame

Pearson correlation matrix.

Examples

>>> import pandas as pd
>>> from neurostatx.utils.preprocessing import (
...     compute_correlation_coefficient)
>>> df = pd.DataFrame({"a": [1, 2, 3], "b": [2, 4, 6]})
>>> corr = compute_correlation_coefficient(df, ".")
>>> corr.loc["a", "b"]
1.0

compute_pcafunction#

View source

def compute_pca(X, n_components)

Fit a PCA and return scores, diagnostics, and the model.

Parameters

Name	Type	Description
X	pd.DataFrame	Input table.
n_components	int	Number of components to keep.

Returns

scoresarray

Transformed data.

pcaPCA

Fitted PCA model.

exp_vararray

Explained variance ratio.

componentsarray

Principal axes.

p_valuefloat

Bartlett sphericity p-value.

kmo_modelfloat

Overall KMO statistic.

Examples

>>> import pandas as pd
>>> from neurostatx.utils.preprocessing import compute_pca
>>> X = pd.DataFrame({"a": [1.0, 2.0, 3.0, 4.0],
...                   "b": [2.0, 1.0, 4.0, 3.0]})
>>> scores, pca, exp_var, components, p_value, kmo = compute_pca(X, 1)
>>> scores.shape
(4, 1)

compute_shapiro_wilk_testfunction#

View source

def compute_shapiro_wilk_test(df)

Run a Shapiro-Wilk normality test on each column.

Parameters

Name	Type	Description
df	pd.DataFrame	Input table.

Returns

wilklist of float

Shapiro-Wilk W statistics.

pvalueslist of float

Corresponding p-values.

Examples

>>> import pandas as pd
>>> from neurostatx.utils.preprocessing import compute_shapiro_wilk_test
>>> df = pd.DataFrame({"a": [1.0, 2.0, 3.0, 4.0]})
>>> wilk, pvalues = compute_shapiro_wilk_test(df)
>>> len(wilk)
1

get_column_indicesfunction#

View source

def get_column_indices(df, column_names)

Return integer positions for the requested column names.

Parameters

Name	Type	Description
df	pd.DataFrame	Input table.
column_names	list of str	Column names to look up.

Returns

indiceslist of int

Column positions. Missing names are skipped.

Examples

>>> import pandas as pd
>>> from neurostatx.utils.preprocessing import get_column_indices
>>> df = pd.DataFrame({"a": [1], "b": [2]})
>>> get_column_indices(df, ["b", "a"])
[1, 0]

merge_dataframesfunction#

View source

def merge_dataframes(dict_df, index, repeated_columns=False)

Join several tables on a shared index column.

Index values must be unique within each table.

Parameters

Name	Type	Default	Description
dict_df	dict	required	Mapping of labels to DataFrames.
index	str	required	Shared column used as the join index.
repeated_columns	bool	False	If True, disambiguate overlapping column names with suffixes. Defaults to False.

Returns

outpd.DataFrame

Joined table.

Examples

>>> import pandas as pd
>>> from neurostatx.utils.preprocessing import merge_dataframes
>>> left = pd.DataFrame({"id": [1, 2], "a": [10, 20]})
>>> right = pd.DataFrame({"id": [1, 2], "b": [3, 4]})
>>> merge_dataframes({"l": left, "r": right}, "id")["a"].tolist()
[10, 20]

plot_distributionsfunction#

View source

def plot_distributions(df, out_folder, context='poster', font_scale=1)

Write a histogram and ECDF for each column in df.

Parameters

Name	Type	Default	Description
df	pd.DataFrame	required	Input table.
out_folder	str	required	Directory where {column}.png files are written.
context	str	'poster'	Seaborn context. Defaults to "poster".
font_scale	float	1	Seaborn font scale. Defaults to 1.

Examples

>>> import pandas as pd
>>> from neurostatx.utils.preprocessing import plot_distributions
>>> df = pd.DataFrame({"a": [1, 2, 3]})
>>> plot_distributions(df, ".")

remove_nansfunction#

View source

def remove_nans(df)

Split a table into rows with missing values and complete rows.

Parameters

Name	Type	Description
df	pd.DataFrame	Input table.

Returns

rows_with_nanspd.DataFrame

Rows that contain at least one NaN.

complete_rowspd.DataFrame

Rows with no missing values.

Examples

>>> import pandas as pd
>>> import numpy as np
>>> from neurostatx.utils.preprocessing import remove_nans
>>> df = pd.DataFrame({"a": [1.0, np.nan], "b": [2.0, 3.0]})
>>> nans, complete = remove_nans(df)
>>> len(complete)
1

rename_columnsfunction#

View source

def rename_columns(df, old_names, new_names)

Rename selected columns and return a copy of the table.

Parameters

Name	Type	Description
df	pd.DataFrame	Input table.
old_names	list of str	Current column names.
new_names	list of str	Replacement column names, same length as old_names.

Returns

new_dfpd.DataFrame

Copy of df with renamed columns.

Examples

>>> import pandas as pd
>>> from neurostatx.utils.preprocessing import rename_columns
>>> df = pd.DataFrame({"a": [1], "b": [2]})
>>> rename_columns(df, ["a"], ["x"]).columns.tolist()
['x', 'b']