about yfcc15m

[konioy]

The YFCC15M data you provided and DeCLIP's YFCC15M label file (yfcc15m_clean_open_data.json) only partially overlap when I compare them. Although they both go by “YFCC15M,” the contents aren't the same — I'm a bit confused.

[konioy]

DeCLIP's YFCC15M link https://github.com/Sense-GVT/DeCLIP/blob/main/docs/dataset_prepare.md#prepare-datasets

[konioy]

my code:

import json

json_urls = set()
with open("yfcc15m_clean_open_data.json", "r", encoding="utf-8") as f:
    for line in f:
        record = json.loads(line)
        json_urls.add(record["url"])


import pandas as pd

count = 0
df_iter = pd.read_csv("yfcc15m_3long_3short_1raw_captions_url.csv", usecols=["Image Path"], iterator=True, chunksize=100000)
for chunk in df_iter:
    for url in chunk["Image Path"].dropna():
        if url in json_urls:
            count += 1

print(count)