why I got a list instead of datasets.arrow_dataset.Column after the map？

[Septemberlemon]

this makes me very confused

[aasimansari1]

This is expected behavior — map() in HuggingFace datasets returns a Dataset, not an individual column object.

After calling .map(), accessing a column via dataset["column_name"] returns a Python list, not a datasets.arrow_dataset.Column. This changed in datasets v2.x.

How to work with the result correctly

# After map, column access returns a list
result = dataset.map(lambda x: {"new_col": x["text"].upper()})
print(type(result["new_col"]))  # <class list>

# To get a proper column object, use .data
import pyarrow as pa
col = result.data["new_col"]  # pyarrow.ChunkedArray

# Or convert to pandas
df = result.to_pandas()
print(df["new_col"])  # pandas Series

If you need the Arrow column object

# Access via the underlying Arrow table
arrow_col = result._data["new_col"]  # ChunkedArray

Why this happens

HuggingFace datasets stores data in Arrow format internally. The [] accessor is a convenience that converts to a Python list for ease of use. The datasets.arrow_dataset.Column type was an older API that is no longer exposed directly.

For most use cases, the list access is what you want. Use .to_pandas() or .to_arrow() if you need more column-level operations.

[Hassanmahmood4]

What you're seeing is expected behavior.

Dataset.map() returns a new Dataset, but when you immediately index a column:

reasoning_dataset.map(generate_conversation, batched=True)["conversations"]

you're no longer working with the Dataset object itself. Instead, you're retrieving the values of the "conversations" column from that dataset.

For a Hugging Face Dataset, column access returns the actual column contents, which behave like a Python list:

mapped_dataset = reasoning_dataset.map(
    generate_conversation,
    batched=True
)

type(mapped_dataset)
# datasets.arrow_dataset.Dataset

type(mapped_dataset["conversations"])
# list

So the difference is:

• mapped_dataset → datasets.Dataset
• mapped_dataset["conversations"] → list

If you were expecting a datasets.arrow_dataset.Column, that may have been the behavior in older versions or in a different context. In current versions of datasets, indexing a dataset by column name materializes that column as a Python list.

If you want to keep working with the dataset structure, assign the result of map() first and operate on the dataset object rather than indexing the column immediately.