Where does data fit in the big picture?

• Defining a problem statement
• Collecting and storing data
  • recording data, downloading data sets, web scraping or using API, data structures and lakes
• Data
• Sampling and Modeling
• Insights, inference, and visualization

Data types

Rows and columns

Rows and columns: Diamond Data Set

• Two use-cases: supervised and unsupervised learning

• Independent variables can be used for unsupervised modeling
• Dependent variables and independent variables can be used together for supervised modeling

Rows and columns

• Rows are linked — some rows belong to the same group
• Are the group means the same?

• Label and categorical independent variable that can be used in a model: city_type
• potential dependent variables: population_mil, rainfall_inches

Categorical variables and dummy encoding

Distributions depend on the data

Terminology of distributions

• Moments: Values from our sample that can help us understand our distribution

• Parameters: The values that define our distribution]

  • Normal: \(\mu\), \(\sigma\)
  • Binomial: \(n\), \(p\)
  • Uniform: \(a\), \(b\)

• Probability Density (Mass) Function (PDF)/(PMF): A description of how likely certain outcomes are in a distribution

• Cumulative Distribution Function (CDF): A description of how much of a distribution is contained up to a certain point

Moments of random variables

• Moments describe a distribution with a set of attributes

• \(\text{E}[X^n]\): general raw moment

• \(\text{E}[X]\): first moment, \(\mu=\frac{\sum{x}}{n}\)

• \(\text{E}[X^2]\): second raw moment (unrefined, so contains first moment information)

• \(\text{E}[X^2]\) - \(\text{E}[X]^2\): second central moment, \(\sigma^2=\frac{\sum{(x-\mu})^2}{n}\)

• For continuous data, we can mathematically represent the second central moment:

$$\int_{\infty}^{-\infty}f(x)x^2dx - \text{E}[X]^2$$ where \(f(x)=PDF\)

Uniform PMF/PDF, CDF

Simulated data: Normal

import random, numpy, math
import statistics as stats, matplotlib.pyplot as plt
import seaborn as sns, pandas as pd
random.seed(50390) # reproducibility
sample_df = numpy.random.normal(
  loc=0, scale = 2, size = 1000
  ) # Normal
sample_df[0:3]

## array([-0.42501843, -0.93878938, -2.88702722])

stats.mean(sample_df)

## -0.051744188640231566

# Solving for sigma using sample variance
math.sqrt(stats.variance(sample_df))

## 1.9919425634418428

Mystery data: Uniform

my_nums = pd.read_csv("my_sample.csv") #sample is uniform
my_nums = my_nums["x"].tolist()
my_nums[0:5]

## [8.74912820779718, 9.16765952832066, 6.329065448371691, 4.0551836041268, 8.59842579229735]

## Data length: 10000

stats.mean(my_nums) #Can we use sample moments to find parameters

## 5.513906104243407

stats.variance(my_nums)

## 6.673169412967319

Calculating Method of Moments: Uniform

\begin{align} \text{E}[\mu] & =\int_{\infty}^{-\infty}\frac{1}{b-a}xdx \\ & = \frac{X^2}{2(b-a)} \\ \frac{X^2}{2(b-a)} ]_a^b& = \frac{b^2}{2(b-a)}-\frac{a^2}{2(b-a)} \\ & = \frac{b^2-a^2}{2(b-a)} \\ \frac{b^2-a^2}{2(b-a)} & = \frac{{(b-a)}(b+a)}{2(b-a)} \\ \text{E}[\mu] & = \frac{a+b}{2} \\ \end{align}

Calculating Method of Moments: Uniform

Checking Method of Moments with the data

print(f"a = {min(my_nums)}, max = {max(my_nums)}")

## a = 1.0011236150749, max = 9.999872184358539

Choosing a representative sample

• Our goal when modeling is to use a sample to draw conclusions about a greater population

• It is important that our sample is representative of our population; Ensuring we have a sufficient sample size is helpful to accomplish this goal

• Sample Moments = Theoretical Moments! If the sample is not representative, this equation will mislead us

• Methods of repeated sampling are also good to utilize, Central Limit Theorem

Central Limit Theorem

dice = [1, 2, 3, 4, 5, 6] #uniform

# Central limit
# Draws from Uniform Distribution becoming normal distribution
rolls = 6
dice_means = [0] * rolls
for i in range(0,rolls):
    this_roll = random.choices(dice,k=rolls)
    dice_means[i] = stats.mean(this_roll)

Using descriptive statistics from a sample

• Measure of central tendency:
  • Mean — arithmetic average
  • Median — middle quantile (based on data sorted)
  • Mode — most common value
• Variance
• Range, IQ range
• Correlation — the foundation of linear regression