Skip to content

Commit 61c9879

Browse files
committed
feat: add map values cleaning step
1 parent 2e3fe96 commit 61c9879

7 files changed

Lines changed: 195 additions & 45 deletions

File tree

API_REFERENCE.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ A technical reference guide to the public classes and functions within the **Arn
88
| :-------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
99
| **Core Class** | [**`ArFrame`**](#arframe), Properties: [`shape`](#shape), [`columns`](#columns), [`dtypes`](#dtypes), [`is_empty`](#is_empty), Methods: [`memory_usage`](#memory_usage), [`preview`](#preview), [`select_columns`](#select_columns), [`select_dtypes`](#select_dtypes) |
1010
| **I/O** | [`read_csv`](#read_csv), [`scan_csv`](#scan_csv), [`write_csv`](#write_csv), [`write_json`](#write_json), [`write_jsonl`](#write_jsonl), [`sniff_delimiter`](#sniff_delimiter) |
11-
| **Cleaning** | [`cast_types`](#cast_types), [`clean`](#clean), [`clip_numeric`](#clip_numeric), [`combine_columns`](#combine_columns), [`drop_columns`](#drop_columns), [`drop_constant_columns`](#drop_constant_columns), [`drop_duplicates`](#drop_duplicates), [`drop_nulls`](#drop_nulls), [`fill_nulls`](#fill_nulls), [`filter_rows`](#filter_rows), [`keep_rows_with_nulls`](#keep_rows_with_nulls), [`normalize_case`](#normalize_case), [`normalize_unicode`](#normalize_unicode), [`rename_columns`](#rename_columns), [`replace_values`](#replace_values), [`round_numeric_columns`](#round_numeric_columns), [`safe_divide_columns`](#safe_divide_columns), [`strip_whitespace`](#strip_whitespace), [`trim_column_names`](#trim_column_names), [`validate_columns_exist`](#validate_columns_exist) |
11+
| **Cleaning** | [`cast_types`](#cast_types), [`clean`](#clean), [`clip_numeric`](#clip_numeric), [`combine_columns`](#combine_columns), [`drop_columns`](#drop_columns), [`drop_constant_columns`](#drop_constant_columns), [`drop_duplicates`](#drop_duplicates), [`drop_nulls`](#drop_nulls), [`fill_nulls`](#fill_nulls), [`filter_rows`](#filter_rows), [`keep_rows_with_nulls`](#keep_rows_with_nulls), [`map_values`](#map_values), [`normalize_case`](#normalize_case), [`normalize_unicode`](#normalize_unicode), [`rename_columns`](#rename_columns), [`replace_values`](#replace_values), [`round_numeric_columns`](#round_numeric_columns), [`safe_divide_columns`](#safe_divide_columns), [`strip_whitespace`](#strip_whitespace), [`trim_column_names`](#trim_column_names), [`validate_columns_exist`](#validate_columns_exist) |
1212
| **Conversion** | [`from_pandas`](#from_pandas), [`to_pandas`](#to_pandas), [`from_arrow`](#from_arrow) |
1313
| **Integration** | [`ArnioPandasAccessor`](#arniopandasaccessor) |
1414
| **Pipeline** | [`pipeline`](#pipeline), [`register_step`](#register_step) |
@@ -349,6 +349,15 @@ Replace values based on a mapping dict.
349349
df = ar.replace_values(df, {"old_value": "new_value"}, column="name")
350350
```
351351

352+
### map_values
353+
354+
Map selected column values while preserving values that are not present in the
355+
mapping.
356+
357+
```python
358+
df = ar.map_values(df, {"M": "Male", "F": "Female"}, subset=["gender"])
359+
```
360+
352361
### round_numeric_columns
353362

354363
Round numeric columns.

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1031,7 +1031,7 @@ Total avg (Read+Strict) 0.077 4.52
10311031

10321032
## 🧰 Cleaning primitives
10331033

1034-
Most operations below run natively in C++. Currently, `filter_rows`, `replace_values` and `standardize_missing_tokens` run via the Python (pandas) backend and may be optimized in C++ later.
1034+
Most operations below run natively in C++. Currently, `filter_rows`, `replace_values`, `map_values` and `standardize_missing_tokens` run via the Python (pandas) backend and may be optimized in C++ later.
10351035

10361036
| Primitive | What it does | Example |
10371037
|:---|:---|:---|
@@ -1054,6 +1054,7 @@ Most operations below run natively in C++. Currently, `filter_rows`, `replace_va
10541054
| `cast_types` | Cast column types with `errors="raise"`, `"coerce"`, or `"ignore"` | `ar.cast_types(frame, {"age": "int64"}, errors="raise")` |
10551055
| `round_numeric_columns` | Round numeric columns (non-numeric columns in subset ignored safely) | `ar.round_numeric_columns(frame, decimals=2)` |
10561056
| `replace_values` | Replace values using a mapping (column or whole-frame). Handles `None`/`NaN`. | `ar.replace_values(frame, {"active": "A", "inactive": "I"}, column="status")` |
1057+
| `map_values` | Map selected column values while preserving unmapped values | `ar.map_values(frame, {"M": "Male"}, subset=["gender"])` |
10571058
| `clean` | Convenience shorthand supporting config dicts | `ar.clean(frame, strip_whitespace={"subset": ["name"]}, drop_nulls=True)` |
10581059
| `safe_divide_columns` | Divide one column by another, handling zero/null denominators | `ar.safe_divide_columns(frame, numerator="revenue", denominator="cost", output_column="ratio")` |
10591060
| `drop_columns_matching` | Drop columns whose names match a regex pattern | `ar.drop_columns_matching(frame, pattern="^temp_")` |

arnio/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
find_fuzzy_duplicates,
2626
hash_columns,
2727
keep_rows_with_nulls,
28+
map_values,
2829
normalize_case,
2930
normalize_minmax,
3031
normalize_unicode,
@@ -167,6 +168,7 @@
167168
"validate_columns_exist",
168169
"filter_rows",
169170
"replace_values",
171+
"map_values",
170172
"normalize_whitespace",
171173
"drop_duplicates",
172174
"find_fuzzy_duplicates",

arnio/cleaning.py

Lines changed: 96 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -2264,6 +2264,91 @@ def _is_null_mapping_key(value):
22642264
return bool(pd.isna(value))
22652265

22662266

2267+
def _apply_value_mapping_to_series(
2268+
series: pd.Series,
2269+
mapping: Mapping[Any, Any],
2270+
*,
2271+
operation: str,
2272+
) -> pd.Series:
2273+
"""Apply scalar value mappings to a Series, including null-like keys."""
2274+
null_key_present = False
2275+
null_replacement = None
2276+
normalized_mapping = {}
2277+
2278+
for k, v in mapping.items():
2279+
if _is_null_mapping_key(k):
2280+
null_key_present = True
2281+
null_replacement = v
2282+
elif is_scalar(k) and not isinstance(
2283+
k, (tuple, list, np.ndarray, pd.Series, pd.Index)
2284+
):
2285+
normalized_mapping[k] = v
2286+
else:
2287+
raise TypeError(
2288+
f"{operation}() does not support non-scalar mapping keys. "
2289+
f"Got key {k!r} of type '{type(k).__name__}'. "
2290+
f"Only scalar values (str, int, float, bool) and null-like keys "
2291+
f"(None, float('nan'), pd.NA, pd.NaT) are supported."
2292+
)
2293+
2294+
original_null_mask = series.isna() if null_key_present else None
2295+
result = series.replace(normalized_mapping) if normalized_mapping else series
2296+
if null_key_present:
2297+
result = result.where(~original_null_mask, null_replacement)
2298+
return result
2299+
2300+
2301+
def map_values(
2302+
frame: ArFrame | pd.DataFrame,
2303+
mapping: Mapping[Any, Any],
2304+
subset: Sequence[str] | None = None,
2305+
) -> ArFrame | pd.DataFrame:
2306+
"""Map selected column values through a user-provided mapping.
2307+
2308+
Values not present in ``mapping`` are preserved. When ``subset`` is
2309+
provided, only those columns are transformed; otherwise the mapping is
2310+
applied to all columns.
2311+
"""
2312+
frame, is_arframe = _validate_frame(frame, allow_pandas=True)
2313+
mapping = _validate_mapping(
2314+
mapping,
2315+
argument_name="mapping",
2316+
allow_empty=False,
2317+
non_mapping_message=(
2318+
"mapping must be a dict-like mapping of {old_value: new_value}, "
2319+
f"not {type(mapping).__name__}."
2320+
),
2321+
)
2322+
2323+
df = to_pandas(frame) if is_arframe else frame.copy(deep=False)
2324+
if subset is None:
2325+
target_columns = list(df.columns)
2326+
else:
2327+
target_columns = _validate_existing_column_sequence(
2328+
subset,
2329+
available_columns=df.columns,
2330+
argument_name="subset",
2331+
reject_duplicates=True,
2332+
missing_error=ValueError,
2333+
missing_message=lambda missing, available: (
2334+
f"Unknown columns in subset: {missing}. Available: {available}"
2335+
),
2336+
)
2337+
2338+
if not target_columns:
2339+
return frame if is_arframe else df
2340+
2341+
result_df = df.copy(deep=False)
2342+
for column in target_columns:
2343+
result_df[column] = _apply_value_mapping_to_series(
2344+
result_df[column],
2345+
mapping,
2346+
operation="map_values",
2347+
)
2348+
2349+
return from_pandas(result_df) if is_arframe else result_df
2350+
2351+
22672352
def replace_values(
22682353
frame: ArFrame | pd.DataFrame,
22692354
mapping: dict,
@@ -2328,50 +2413,19 @@ def replace_values(
23282413
f"Column '{column}' not found. Available columns: {available}"
23292414
)
23302415

2331-
# Normalize mapping and separate null-key handling because NaN != NaN
2332-
null_key_present = False
2333-
null_replacement = None
2334-
normalized_mapping = {}
2335-
2336-
for k, v in mapping.items():
2337-
# Handle scalar null-like keys safely without evaluating
2338-
# tuple/list/array-like objects in boolean context.
2339-
if _is_null_mapping_key(k):
2340-
null_key_present = True
2341-
null_replacement = v
2342-
# Exclude tuple/list/ndarray/series/index keys which pandas.replace
2343-
# does not support and can raise confusing errors (e.g. operand
2344-
# length mismatch). Treat strings and true scalars as valid keys.
2345-
elif is_scalar(k) and not isinstance(
2346-
k, (tuple, list, np.ndarray, pd.Series, pd.Index)
2347-
):
2348-
normalized_mapping[k] = v
2349-
else:
2350-
raise TypeError(
2351-
f"replace_values() does not support non-scalar mapping keys. "
2352-
f"Got key {k!r} of type '{type(k).__name__}'. "
2353-
f"Only scalar values (str, int, float, bool) and null-like keys "
2354-
f"(None, float('nan'), pd.NA, pd.NaT) are supported."
2355-
)
2356-
23572416
if column:
2358-
s = df[column]
2359-
original_null_mask = s.isna() if null_key_present else None
2360-
if normalized_mapping:
2361-
s = s.replace(normalized_mapping)
2362-
if null_key_present:
2363-
# Replace only values that were already null before replacement so
2364-
# null-valued mapping results remain real nulls.
2365-
s = s.where(~original_null_mask, null_replacement)
2366-
df[column] = s
2417+
df[column] = _apply_value_mapping_to_series(
2418+
df[column],
2419+
mapping,
2420+
operation="replace_values",
2421+
)
23672422
else:
2368-
original_null_mask = df.isna() if null_key_present else None
2369-
if normalized_mapping:
2370-
df = df.replace(normalized_mapping)
2371-
if null_key_present:
2372-
# Replace only values that were already null before replacement so
2373-
# null-valued mapping results remain real nulls.
2374-
df = df.where(~original_null_mask, null_replacement)
2423+
for current_column in df.columns:
2424+
df[current_column] = _apply_value_mapping_to_series(
2425+
df[current_column],
2426+
mapping,
2427+
operation="replace_values",
2428+
)
23752429

23762430
return from_pandas(df) if is_arframe else df
23772431

arnio/pipeline.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -940,6 +940,7 @@ def pipeline(
940940
register_step("rename_columns_matching", cleaning.rename_columns_matching)
941941
register_step("safe_divide_columns", cleaning.safe_divide_columns)
942942
register_step("replace_values", cleaning.replace_values)
943+
register_step("map_values", cleaning.map_values)
943944
_BUILTIN_PYTHON_STEP_REGISTRY.update(_PYTHON_STEP_REGISTRY)
944945

945946

tests/test_cleaning.py

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3446,6 +3446,88 @@ def test_replace_values_pandas_dataframe_input_returns_dataframe(self):
34463446
assert df["status"].tolist() == ["active", "inactive"]
34473447

34483448

3449+
class TestMapValues:
3450+
def test_map_values_applies_mapping_to_subset(self):
3451+
frame = ar.from_pandas(
3452+
pd.DataFrame(
3453+
{
3454+
"gender": ["M", "F", "X"],
3455+
"status": ["M", "active", "inactive"],
3456+
}
3457+
)
3458+
)
3459+
3460+
result = ar.map_values(
3461+
frame,
3462+
{"M": "Male", "F": "Female"},
3463+
subset=["gender"],
3464+
)
3465+
df = ar.to_pandas(result)
3466+
3467+
assert list(df["gender"]) == ["Male", "Female", "X"]
3468+
assert list(df["status"]) == ["M", "active", "inactive"]
3469+
3470+
def test_map_values_preserves_unmapped_values(self):
3471+
frame = ar.from_pandas(pd.DataFrame({"status": ["new", "done", "held"]}))
3472+
3473+
result = ar.map_values(frame, {"new": "open"}, subset=["status"])
3474+
df = ar.to_pandas(result)
3475+
3476+
assert list(df["status"]) == ["open", "done", "held"]
3477+
3478+
def test_map_values_supports_null_keys_and_pd_na_replacements(self):
3479+
frame = ar.from_pandas(
3480+
pd.DataFrame({"status": ["active", "missing", None, pd.NA]})
3481+
)
3482+
3483+
result = ar.map_values(
3484+
frame,
3485+
{"missing": pd.NA, pd.NA: "unknown"},
3486+
subset=["status"],
3487+
)
3488+
df = ar.to_pandas(result)
3489+
3490+
assert df.loc[0, "status"] == "active"
3491+
assert pd.isna(df.loc[1, "status"])
3492+
assert df.loc[2, "status"] == "unknown"
3493+
assert df.loc[3, "status"] == "unknown"
3494+
3495+
def test_map_values_rejects_unknown_subset_column(self):
3496+
frame = ar.from_pandas(pd.DataFrame({"status": ["active"]}))
3497+
3498+
with pytest.raises(ValueError, match="Unknown columns in subset"):
3499+
ar.map_values(frame, {"active": "A"}, subset=["missing"])
3500+
3501+
def test_map_values_dataframe_input_returns_dataframe(self):
3502+
df = pd.DataFrame({"gender": ["M", "F"], "score": [1, 2]})
3503+
3504+
result = ar.map_values(df, {"M": "Male"}, subset=["gender"])
3505+
3506+
assert isinstance(result, pd.DataFrame)
3507+
assert result["gender"].tolist() == ["Male", "F"]
3508+
assert result["score"].tolist() == [1, 2]
3509+
assert df["gender"].tolist() == ["M", "F"]
3510+
3511+
def test_map_values_pipeline_integration(self):
3512+
frame = ar.from_pandas(pd.DataFrame({"gender": ["M", "F", "M"]}))
3513+
3514+
result = ar.pipeline(
3515+
frame,
3516+
[
3517+
(
3518+
"map_values",
3519+
{
3520+
"mapping": {"M": "Male", "F": "Female"},
3521+
"subset": ["gender"],
3522+
},
3523+
)
3524+
],
3525+
)
3526+
df = ar.to_pandas(result)
3527+
3528+
assert list(df["gender"]) == ["Male", "Female", "Male"]
3529+
3530+
34493531
class TestRoundNumericColumns:
34503532
def test_round_subset_missing_column_raises_clear_error(self):
34513533
import pandas as pd

0 commit comments

Comments
 (0)