AIML4OS WP13 – ISTAT Use Case 2

Synthetic Data Generation and Utility–Privacy Trade-off Assessment

This repository contains the Python code developed by Istat within Work Package 13 (WP13) of the AIML4OS project for the Italian Use Case 2.

The use case focuses on the development of a controlled experimental framework to assess the trade-off between:

• utility, i.e. the ability of synthetic data to preserve relevant statistical and analytical information from the original data;
• privacy protection, i.e. the extent to which synthetic data reduce the risk of disclosing information about the original records.

The repository is intended to support reproducibility, transparency and reuse of the methodology developed within AIML4OS.

---

1. Context and objective

Synthetic data are increasingly considered as a possible solution for enabling data sharing, methodological experimentation and collaborative research while reducing privacy risks.

However, generating synthetic data is not sufficient by itself. It is also necessary to evaluate whether the generated data:

1. preserve enough information to be useful for statistical or machine learning tasks;
2. avoid reproducing individual-level information too closely;
3. provide a measurable balance between analytical utility and privacy protection.

For this reason, the ISTAT Use Case 2 adopts a controlled experimental setting. A synthetic “real” dataset is generated first, with known structure and controlled relationships between variables. This dataset is then used as the reference data source for generating synthetic datasets and evaluating both utility and privacy risk.

The general workflow is:

1. build a controlled reference dataset;
2. generate synthetic data using different methods;
3. evaluate utility through classification tasks;
4. evaluate privacy risk through attack scenarios;
5. compare results across methods and experimental settings.

---

2. Repository structure

The repository is organized as a sequence of Jupyter notebooks, each corresponding to a specific step of the workflow.

AIML4OS_WP13_ISTAT_UC2/
│
├── notebooks/
│   ├── Step_1_Generate real dataset.ipynb
│   ├── Step_2a_Synthetic_data_generation_SDV.ipynb
│   ├── Step_2b_Synthetic_data_generation_Inc-XGB.ipynb
│   ├── Step_3_Classification_Task.ipynb
│   ├── Step_4a_First_Privacy_Attack_with_distance_metrics.ipynb
│   └── Step_4b_Second_Privacy_Attack_with_ML.ipynb
│
├── sspcloud/
│   ├── download_data.sh
│   ├── download_notebook.sh
│   ├── init-trainees.sh
│   └── restore_environment.sh
│
├── chapters/
├── images/
├── resources/
├── pyproject.toml
└── README.md

The folder notebooks/ contains the core analytical workflow.

The folder sspcloud/ contains shell scripts useful for setting up the working environment and downloading data/notebooks in the SSPCloud/Onyxia environment.

The other folders are related to the AIML4OS Quarto training material template and documentation structure.

---

3. Workflow description

Step 1 – Generation of the reference dataset

Notebook:

notebooks/Step_1_Generate real dataset.ipynb

This notebook builds the reference dataset used in the experiment.

The dataset is artificially generated but designed to resemble a realistic statistical dataset. Some demographic distributions, such as age, gender and civil status, are derived from publicly available Istat demographic information.

The generated dataset includes variables such as:

• demographic variables;
• municipality-related variables;
• physical activity;
• genetic predisposition;
• a target variable representing a four-class diagnosis;
• derived score variables used to control the relationship between explanatory variables and the target.

Although the dataset is artificial, it is treated in the experiment as the “real” dataset. This allows the privacy and utility assessment to be performed in a controlled and reproducible way.

---

Step 2a – Synthetic data generation using SDV

Notebook:

notebooks/Step_2a_Synthetic_data_generation_SDV.ipynb

This notebook applies synthetic data generation methods based on the Synthetic Data Vault (SDV) ecosystem.

The aim is to generate synthetic versions of the reference dataset using model-based approaches and then compare their analytical behaviour with the original data.

Depending on the specific configuration, this step may include generative models such as CTGAN, TVAE or other SDV-compatible synthesizers.

---

Step 2b – Synthetic data generation using Inc-XGB

Notebook:

notebooks/Step_2b_Synthetic_data_generation_Inc-XGB.ipynb

This notebook implements the synthetic data generation process based on an incremental XGBoost-related approach.

The rationale is to generate synthetic records by modelling the variables sequentially or conditionally, preserving relevant multivariate relationships among variables.

This step provides an alternative to the SDV-based generators, allowing comparison across different synthetic data generation strategies.

---

Step 3 – Utility assessment through classification

Notebook:

notebooks/Step_3_Classification_Task.ipynb

This notebook evaluates the utility of the generated synthetic datasets.

The main idea is to test whether synthetic data preserve enough information to support a predictive task comparable to the one performed on the reference dataset.

Typical classification models considered in this framework include:

• Logistic Regression;
• Random Forest;
• Multi-Layer Perceptron;
• XGB.

The classification task focuses on the target variable generated in Step 1. Utility is assessed by comparing model performance across real and synthetic data under different train/test configurations.

---

Step 4a – Privacy attack based on distance metrics

Notebook:

notebooks/Step_4a_First_Privacy_Attack_with_distance_metrics.ipynb

This notebook implements a first privacy-risk assessment based on distance metrics.

The underlying idea is to evaluate whether synthetic records are too close to original records. If a synthetic dataset contains records that are very similar to individual records in the reference dataset, this may indicate a higher risk of memorization or disclosure.

This step may include:

• nearest-neighbour searches;
• comparison between real and synthetic records;
• top-k similarity analysis;
• distance-based metrics suitable for mixed-type data.

The objective is not only to measure closeness, but also to understand whether the synthetic generator is learning general population patterns or reproducing individual-level information too precisely.

---

Step 4b – Privacy attack based on machine learning

Notebook:

notebooks/Step_4b_Second_Privacy_Attack_with_ML.ipynb

This notebook implements the second privacy attack, based on a machine-learning approach.

The attack does not directly compare original and synthetic records through distance metrics. Instead, it builds a set of meta-features that describe how a classification model behaves when it is trained and tested on different combinations of real and synthetic data.

These meta-features summarize the predictive behaviour of the models and are then used to train an attack classifier. The goal is to assess whether the synthetic data preserve only general statistical patterns or whether they contain signals that make them too informative about the original data.

---

4. Expected outputs

The workflow produces intermediate and final outputs that may include:

• generated reference datasets;
• synthetic datasets obtained with different generators;
• classification performance indicators;
• privacy attack results;
• tables and figures for comparing utility and privacy risk;
• evidence supporting the evaluation of the utility–privacy trade-off.

The exact output files depend on the execution environment and on the configuration used in each notebook.

---

5. How to run the notebooks

A recommended execution order is:

1. Step_1_Generate real dataset.ipynb
2. Step_2a_Synthetic_data_generation_SDV.ipynb
3. Step_2b_Synthetic_data_generation_Inc-XGB.ipynb
4. Step_3_Classification_Task.ipynb
5. Step_4a_First_Privacy_Attack_with_distance_metrics.ipynb
6. Step_4b_Second_Privacy_Attack_with_ML.ipynb

The notebooks are designed to be executed sequentially, since the outputs of one step are used as inputs for the following steps.

Before running the workflow, users should check:

• the expected input and output folders;
• the Python environment;
• the availability of required packages;
• the execution path used by the notebooks;
• whether the data download scripts in sspcloud/ are needed for the selected environment.

---

6. Methodological notes

This use case is based on a controlled experimental framework. This means that the “real” dataset is not an actual confidential microdata file, but an artificially generated dataset whose structure is known by design.

This choice has several advantages:

• it allows the experiment to be fully reproducible;
• it avoids the use of confidential data;
• it makes it possible to control the relationship between input variables and the target variable;
• it supports a clearer interpretation of utility and privacy results;
• it provides a safe environment for testing privacy attacks.

The framework can be adapted to other datasets, provided that the data structure, the target variable and the privacy attack scenarios are properly redefined.

---

7. Attribution of code development

The following table can be used to document the contribution of each colleague to the different parts of the workflow.

Step	Notebook / component	Main activity	Contributor(s)
Step 1	Step_1_Generate real dataset.ipynb	Generation of the controlled reference dataset and definition of the target variable	Fabio Spagnuolo (Istat)
Step 2a	Step_2a_Synthetic_data_generation_SDV.ipynb	Synthetic data generation using SDV-based methods	Massimo De Cubellis (Istat)
Step 2b	Step_2b_Synthetic_data_generation_Inc-XGB.ipynb	Synthetic data generation using Inc-XGB approach	Fabio Spagnuolo (Istat)
Step 3	Step_3_Classification_Task.ipynb	Utility assessment through classification models	Donatella Papa (Istat)
Step 4a	Step_4a_First_Privacy_Attack_with_distance_metrics.ipynb	Privacy-risk assessment using distance-based attacks	Donatella Papa (Istat) and Fabio Spagnuolo (Istat)
Step 4b	Step_4b_Second_Privacy_Attack_with_ML.ipynb	Privacy-risk assessment using machine-learning-based attacks	Donatella Papa (Istat)

---

8. Suggested citation / acknowledgement

When reusing or referring to this material, please acknowledge:

Istat contribution to AIML4OS Work Package 13 – Synthetic Data, Use Case 2: Synthetic Data Generation and Utility–Privacy Trade-off Assessment Framework.

A more formal citation can be added when a project deliverable, working paper, conference contribution or journal article associated with this repository becomes available.

---

9. Disclaimer

This repository is intended for research, methodological experimentation and training purposes within the AIML4OS project.

The dataset generated in the workflow is artificial and should not be interpreted as an official Istat statistical release. The results produced by the notebooks depend on the experimental configuration and should be interpreted in the context of the methodological framework described above.

---