Data Shapes Our World
Data plays a central role in today's world as it informs decision-making, drives innovation, enables risk management, advances research and development in medicine, geosciences, climate science, finance, and many more fields and sectors. We all share data constantly, either willingly or unwillingly, by leaving a data trace on the devices we use and in the web, or by participating in a research or marketing studies and filling out a questionnaire.
It is thus also your data that impacts decision making, research and innovation, or advertisement, which, again, have an effect on how you live your life. So data is not just relevant for data scientists and statisticians: it is about you.
However, data also has its limitations and shortcomings. It can, for example, contain biases that make the conclusions drawn from data flawed, or sensitive information about individuals that violates their right to privacy.
In this article, I provide an overview of how and under which circumstances fake data, so-called synthetic data, can address the above mentioned shortcomings of (real-world) data and explain what synthetic data is and how it is generated.
The Power and Limits of Real Data
Tech giants like Google or Meta have become among the most valuable companies in the world today because of data: They put huge efforts into collecting as much data from their users as possible, to learn exactly who they are and what they might want to buy, in order to sell this information in the form of personalised advertising services to other companies.
Data has huge potential: When analysed properly, with statistics and machine learning (ML) methods, knowledge can be extracted from data. When curated and cleaned properly, data can be used to train ML algorithms and AI to automate, and in the best case, improve our lives.
When Data Is Hard to Get
Consider, for example, a data set collected during a clinical study on the effect of a new cancer treatment. The data set is in the form of a table, where one row corresponds to a cancer patient participating in the study. The columns of the data table contain demographic features of the patient, e.g., age, sex, etc., and the clinical parameters, e.g., BMI, cancer-specific biomarkers, indicating the patient’s well-being and the treatment response. A statistician analysing the data set can distill the information in the data to, for example, come to the conclusion: "The proposed treatment reduces the probability of remission of the cancer by 11%.“
As another example, consider a data set that consists of images of normal and abnormal moles. Each image was analysed by a dermatologist and labeled according to whether it shows a normal mole, a suspicious mole or skin cancer, also called melanoma. This image data set can be used to train a ML algorithm to learn to distinguish between a normal, abnormal, or cancerous mole with high accuracy. Such a trained algorithm can then be deployed to support dermatologists in everyday cancer screening, making it quicker and more accurate.
Small, Expensive, and Private
In both examples, the data is highly resource-intensive to collect and curate: A new cancer treatment itself may be very costly to produce and administer to the patients, it is time-consuming to recruit and follow up patients, and only very few patients may even fulfil the requirements for participating in the treatment effect study.
It is also very laborious for a dermatologist to analyse and annotate hundreds of mole images—this is also time they cannot spend on treating patients. As a result, the cancer treatment data set, as well as the labeled mole image data set, will only contain a small (up to 100) to medium (100 to 1000) number of observations.
A small data set, on the other hand, limits the strength of study findings in the first example. And it may lead to an unstable and inaccurate algorithm in the second example.
Additionally, the data in both examples is data of individuals, and therefore highly sensitive and private. For this reason, neither the cancer treatment data nor the mole image data can just be made publicly available online. This prohibits other researchers from replicating and verifying the study findings, conducting their own analysis, or training their own algorithms on the data.
Synthetic Data
Fake data, so-called synthetic data, can solve both of these problems: Synthetic data can supplement the small data sets in our examples to make statistical findings stronger, and algorithms trained on it more reliable. And synthetic data can be published instead of the private real data for other researchers to use.
Synthetic data is fake, artificial data. This means that a row in the synthetic data set has no correspondence in the real world: In the cancer treatment example, a row in the synthetic table does not correspond to any real existing patient, and a synthetic image of a mole, in the second example, is not a photograph of a real person’s mole. While this might sound confusing, this will become clearer in the following.
How Synthetic Data Is Generated
Synthetic data is data generated from ML models and algorithms: First, the real data set is given to the ML model as an input. The ML model then learns the statistical properties of the real data: In the cancer treatment study data, the model learns, for example, that the study participants are between 29 and 67 years old, and most of them are in their fifties.
The model also learns dependencies in the real data, e.g., that older patients in the study tended to benefit less from the new cancer treatment than younger study participants. After the model has analysed the real data closely, the model produces truly new data with the properties it has learnt from the real data - this is the synthetic data.
Thus, in the synthetic cancer treatment data, the synthetic patients are again between about 29 and 67 years old, and also, there, the treatment shows a higher effect on younger (synthetic) patients. And in the mole image example, the synthetic mole image data will not contain images of squared, turquoise moles, but again realistically looking, approximately round and brown moles.
When Fake Data Fails
But what if, in contrast to the examples above, the ML model that generates the synthetic data does not manage to learn the properties of and dependencies in the real data properly? Then the resulting synthetic data does not fulfil its purpose: Such low-quality synthetic data is useless for supplementing a small real data set or replacing private real data.
Quality Is Everything
The quality of data is not only crucial in the synthetic data setting. The importance of high-quality, annotated (synthetic) data on ML and AI models in particular, and, through ML and AI, on scientific progress in general, can hardly be overestimated.
Take, for example, the MNIST data set [2]. This is a familiar name if you have a background in ML. The MNIST data set is a data set with about 70,000 grey-scale images of hand-written digits from 0 to 9, with labels indicating which digit each image depicts (some handwriting is just unreadable). It is a large data set, clean, fully annotated, balanced in the number of images per digit, diverse in the handwriting style, and not sensitive or private in any way.
Because of this, it is publicly available online for anyone to download and work on. And it was, and still is, heavily used to develop, test, and compare new ML and AI models. The MNIST data set enabled tremendous progress in ML. This is because without good and diverse data, it is impossible to assess and test a newly developed ML model. So high-quality data sets play a key role in the success story of ML and AI.
But the high quality of synthetic data is not a given fact. It depends on two main factors: the model that generates the synthetic data and the underlying real data.
Privacy by Design
In the cancer treatment data set, each entry corresponds to a real patient. For the synthetic data to successfully replace the real data, it needs to be private. This means that the synthetic data must not reveal specific information about any patient in the real data set that allows identification of the patient. Synthetic data can only be private if the model generating it is private. This is because the synthetic data is, in some sense, only a realisation of the generative model itself.
Let us assume that our generative model is an algorithm that, when given a real data set, adds another column, „favourite colour“, to the data table and fills it randomly with values „red“, „blue“, „green“, and „yellow“. The resulting synthetic cancer treatment data is equivalent to the real data except for this new „favourite colour“ column.
Thus, the generative model and the resulting synthetic data provide no privacy at all for the patients in the real data, and the synthetic data is useless for being made publicly available instead of the private real data. This is a rather silly example, but it illustrates a real-world phenomenon well: Generative models and algorithms that memorize observations of the real data (here, information of single patients) to exactly reproduce them in the synthetic data (see, for example, [3,4]).
Luckily, there are several principled and clever ways to better ensure the privacy of the generative model and synthetic data. One strategy is to take privacy considerations on the specific real data set into account.
This could, for example, be that the generative model specifically protects sensitive columns in the real data while spending less effort on protecting non-sensitive features [5]. Another strategy, called „differential privacy“ [6], simply put, disturbs the real data, which makes it harder to recognize single observations.
Bias In, Bias Out
The real data, on the other hand, plays an equally important role for the quality of the synthetic data. As the synthetic data reproduces properties of the real data, the synthetic data will not be of better quality than the real data it is based on (unless extra information is added in the synthetic data-generating process from an additional data source). So the real data needs to be informative and diverse for the synthetic data to be informative and diverse.
Moreover, any discrimination in the data collection process and any biases in the real data will be replicated in the synthetic data if they are not corrected or removed. Take, for example, the mole image data set again. In the data set, only 2% of the images show moles of people of colour—a clear underrepresentation of ethnicities that will be replicated in the synthetic data if not accounted for.
A ML algorithm trained on this biased synthetic data has considerably fewer images to learn from for people of colour. Consequently, the algorithm will perform worse in distinguishing normal moles from skin cancer for these ethnicities and, when deployed in a clinical setting to support melanoma screening, it will misclassify and overlook melanomas in people of colour more often than for ethnically white individuals.
Why It’s Worth the Effort
It requires effort and resources to ensure high quality of data—both real and synthetic. But, having in mind that all biases and flaws of our (synthetic) data will be built into ML and AI models that impact how we live and support and take over decisions more and more—it is worth the effort!
References
[1] Arthur, C. (2013). „Tech giants may be huge, but nothing matches big data“. The Guardian. https://www.theguardian.com/technology/2013/aug/23/tech-giants-data. Accessed: 09.09.2025.
[2] LeCun, Y. (1998). The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/.
[3] Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2019). The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19) (pp. 267-284).
[4] Hartley, J., Sanchez, P. P., Haider, F., & Tsaftaris, S. A. (2023). Neural networks memorise personal information from one sample. Scientific Reports, 13(1), 21366.
[5] Griesbauer, E., Czado, C., Frigessi, A., & Haff, I. H. (2025, April). TVineSynth: A Truncated C-Vine Copula Generator of Synthetic Tabular Data to Balance Privacy and Utility. In International Conference on Artificial Intelligence and Statistics (pp. 3511-3519). PMLR.
[6] Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and trends® in theoretical computer science, 9(3–4), 211-407.