Processing large amounts of data is not only complex, but also time-consuming. This article is about the theory and notebooks for data analysis.
Exploratory data analysis (EDA) is a statistical approach that cleans, examines, and analyzes a data set. This method of investigation allows you to learn more about your data and what you can learn from it using aggregated statistical metrics and graphical tools.
This approach is an important part of data science projects. The preparation of data for use in a model is often underestimated, after all, data is not always available in the way you need it. The effort required for preparation is often 80 percent of the time required for a project.
In this article we describe what needs to be considered when collecting, cleaning and transforming data. After the theory, we deal with so-called notebooks for professional data analysis.
Data is not just data
First of all, a distinction is made between qualitative and quantitative data. Data from interviews is of a qualitative nature, data from measurements or surveys is of a quantitative nature. For example, if you are asked which party you would vote for if there were federal elections next Sunday, then the possible answers would be countable and therefore quantitative. But if the question were asked why you want to vote for a certain party without giving you possible answers, then the evaluator could paraphrase and categorize the answers and thus make them countable, but programs could not do this easily.
Obviously, it is difficult to analyze such data from interviews. Unless it is a text analysis using computational linguistic methods. The situation is different with quantitative data, which is usually available in such a way that it can be evaluated automatically without text analysis.
