There are plenty of data science algorithms, machine learning models, and data science predictive frameworks, but a fundamental engine (core fundamentals) behind all of them is statistics for data science.
Learning to apply statistics effectively in data science will help you to grow into a professional analyst who can deliver actionable business value and not only code up solutions. This guide is meant to introduce key concepts, show useful techniques, and offer examples, as well as give you the step-by-step process for gaining fluency in statistics for data science.
Why do data traps occur? They occur. Why do data traps occur?
Overfitting models to noise rather than true signals.
This causes the business to make wrong decisions because it confuses correlation with causation.
P-Value Misuse in A/B Testing.
Selecting to work with imbalanced datasets and selecting the wrong evaluation metric (e.g. accuracy).
Statistics in Data Science is crucial in various stages of the Data Science lifecycle, such as data cleaning, feature engineering, selecting models, and giving recommendations.
The Two Pillars: Descriptive vs. Inferential Statistics
So for the analysis of the data of a data science project, first there has to be statistics to analyze the data. To structure the analysis of the data of a data science project, we work with the two types of statistics: descriptive statistics and inferential statistics.
┌─────────────────────────────────────────┐
│ Statistics for Data Science │
└────────────────────┬────────────────────┘
│
┌──────────────────────────┴──────────────────────────┐
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ Descriptive Statistics │ │ Inferential Statistics │
├─────────────────────────┤ ├─────────────────────────┤
│ • Central Tendency │ │ • Hypothesis Testing │
│ • Dispersion & Variance │ │ • Confidence Intervals │
│ • Skewness & Kurtosis │ │ • Regression & ANOVA │
└─────────────────────────┘ └─────────────────────────┘
1. Descriptive Statistics for Data Science
Data Science with Descriptive Statistics for Data Science—first! Before building complex machine learning models, you must summarize and explore your dataset.
This post is the first of two parts: Data Science with Descriptive Statistics for Data Science.
Measures of Central Tendency
The central tendency of a data set is the middle value of a dataset. Numbers are used to represent the numbers of the data set. These measures of central tendency are used to provide insight on the data distribution in data science.
- Mean: The arithmetic average. Sensitive to extreme outliers.
- Median: Middle value, e.g. in a sorted array, robust to outliers.
- Mode: The most frequently occurring value. In many cases of data science, the
Measures of Dispersion & Spread
Another measure for assessing a data set is how it distributes around the center:
Range: The difference between maximum and minimum values.
Variance ($\sigma^2$ or $s^2$): Average of the squared differences from the mean for a set of numbers (data).
The standard deviation (σ or s) is the square root of variance. It therefore expresses the spread of the data in the same units as were used for the measurement.
Interquartile Range (IQR): IQR is equal to $Q_3 - Q_1$ or $IQR = Q_3 - Q_1$. IQR is used in the calculation of the boxplot and can be used to find outliers in a data set.
Boxplot & Quartile Distribution
Outlier Q1 Median (Q2) Q3 Outlier
o |-------|==============|=============|-------| o
^ <----------- IQR -----------> ^
Min Whisker Max Whisker
2. Statistical Analysis for Data Science & Inferential Techniques
The standard deviation (σ or s) is the square root of variance. Therefore, the standard deviation is an expression for the spread of a set of numbers in the same units in which the numbers were measured.
Hypothesis testing: Testing whether an observed effect is real or just a chance finding.
Null Hypothesis ($H_0$): No effect; no difference; no relationship.
The alternative hypothesis is a statement of what you believe to be true about the population. This can include a statement of what value the true population characteristic(s) would have if there were any difference/relationship (i.e. greater than/less than).
p-Value: The probability of obtaining results as extreme or even more extreme than the results observed in the sample, assuming that there is no real relationship or difference in the population. assumption that the null hypothesis is correct. For typical significance levels (alpha) equal to 0.05, the null hypothesis will be rejected if the p-value is smaller than 0.05.
Type II Error ($\beta$): The false negative error, where you failed to reject the null hypothesis when it is actually false.
Confidence Intervals.
A confidence interval states an estimated range of values and an associated confidence level, e.g. 95%. Using confidence intervals to state the customer churn rate instead of stating the estimated customer churn rate of 12% would result in a statement of 95% 13.5%) for the true population churn rate.
Core Concepts in Probability and Statistics for Data Science
Core Concepts in Probability and Statistics for Data Science
Probability measures uncertainty. In data science many machine learning algorithms make predictions based on probability values. Such algorithms are for instance Naive Bayes and deep probabilistic neural networks.
Key Probability Distributions
Most Data Scientists learn to apply a variety of different probability distributions to represent different types of data, and subsequently apply the appropriate algorithm(s) and model assumption(s) to handle each distribution in turn.
Bayes' Theorem and Conditional Probability
Conditional Probability: This measures the likelihood of an event occurring given another event has occurred. So for event A to occur given event B has occurred, we would measure P(A|B).
$$P(A\vert{}B) = \frac{P(B\vert{}A) \cdot P(A)}{P(B)} $$
Where:
$P(A\vert{}B)$ is the posterior probability (updated belief).
$P(B\vert{}A)$ is the likelihood of observing evidence given $A$.
$P(A)$ is the prior probability (initial belief).
$P(B)$ is the marginal probability (total evidence likelihood).
Bayes’ theorem is used in spam filters, in medical diagnosis, in recommendation systems as well as in Bayesian optimization for hyperparameter settings of parameters of learning algorithms.
Practical Application: Statistics in Machine Learning
Statistical principles directly inform practical model development tasks:
- Feature Engineering & Selection: The correlation between predictors can be investigated using correlation matrices (Pearson and Spearman) in order to select relevant features.
- K-Fold Cross-Validation: This is a statistical sampling method for model validation in order to find out how a model would behave if it were to encounter data that it has not seen before.
- A/B Testing in Product Analytics: In order to verify the increase in conversion rates for different website variations, two-sample t-tests or Z-tests are applied.
- Evaluation Metrics: For binary outcome variables, evaluation metrics such as precision, recall and F1 score can be calculated using joint and conditional probabilities to create confusion matrices, ROC-AUC curves, etc. to assess model performance.
Key Takeaways
Core Roadmap to Mastery
Descriptive Stats Probability Inferential Stats ML Applications
┌───────────────┐ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Mean, Median, │ ──► │ Bayes, Normal │ ──► │ Hypothesis │ ──► │ A/B Testing, │
│ Std Dev, IQR │ │ Distributions │ │ Testing & p-v │ │ Feature Select│
└───────────────┘ └───────────────┘ └───────────────┘ └───────────────┘
When starting to work with data it is good to first explore the data in order to get a grip on the data before fitting models to it.
All models have assumptions so verify that your data fits the model’s assumptions (normality, homoscedasticity, and independence of observations).
Focus on business impact: Make sure that the statistical significance you find translates into real business significance.
Frequently Asked Questions (FAQs)
1. How much statistics is required to start a career in data science?
Descriptive statistics, basic probability, linear regression, and hypothesis testing are sufficient for starting a career in data science The level of required knowledge about statistics depends on the specific data science role (Quantitative Analyst, Research Scientist, etc.). For instance, more advanced knowledge of time-series analysis, Bayesian statistics, and quantitative analyst, is required by a Research Scientist compared to a Business Development Manager.
2. What is the main difference between Descriptive and Inferential Statistics?
There is a distinction between Descriptive Statistics that describes the data from a given dataset using corresponding statistics (like the mean/median/standard deviation etc. of a given set of data) and Inferential Statistics, that, based on the corresponding statistics from a sample of data, makes statements about the wider population. These statements are based on the data gathered and they also describe this data but in addition they can predict observations from the wider population which were not part of the sample.
3. What programming languages are best for statistical analysis?
Python and R are typically the programming languages of choice for a data scientist who does a lot of statistical analysis. The libraries and packages for Python for statistical analysis are SciPy, Statsmodels, NumPy and Pandas. For R, there are many pure statistical models and there are also visualization packages, especially for R, ggplot2.
4. Why is the Normal Distribution so central in data science?
Many natural phenomena and behaviorally generated measurements are naturally distributed in a Normal Distribution. Additionally, a very useful Theorem called the Central Limit Theorem (CLT) proves that the distribution of the sample means for random samples from virtually any underlying population distribution (even not Normally distributed) approaches a Normal Distribution as the sample size increases.
5. How do probability and statistics differ in data science?
Probability is typically used for deductive reasoning (i.e. working from known models and parameters to make predictions about future events). In contrast, statistics is used for inductive reasoning (i.e. analyzing data to make conclusions and predictions about unknown populations and parameters).
Do visit our channel to know more: SevenMentor
Author:-
Pooja Kulkarni
Pooja Kulkarni
Expert trainer and consultant at SevenMentor with years of industry experience. Passionate about sharing knowledge and empowering the next generation of tech leaders.