IMEN891K / IDSC501 / AXGS-501 · Data Engineering · Fall 2026

3. Data Exploration

Sangwoo Mo · POSTECH
Industrial & Management Engineering

Where We ArePart I (Lec 2–5): collection, exploration, and preprocessing

Outline: Data Exploration

Three Ways to Read One TableNumbers, pictures, relations: what each of today’s three parts produces

Descriptive Analysis

Central Tendency

Recap: Mean, Median, ModeThree answers to “the typical value”

meanx̄ = (1/n) ∑_i x_i|median(x_(n/2) + x_(n/2+1)) / 2|modeargmax_x count(x)

From 9, 3, 1, 8, 3, 6: mean 5, median 4.5, mode 3

Skew and the Three CentersIn this right-skewed example, the mean lies farther into the tail

nonparametric skew = (x̄ − median) / sa mean–median comparison, not a complete description of shape

Here, mode < median < mean; check the distribution before generalizing

Robust StatisticsEvery summary comes in a version built to ignore the far end

The split is not about the mean: every summary number has both versions

Variability

Recap: Standard DeviationThe 68–95–99.7 rule

s = √( (1/(n−1)) ∑_i (x_i − x̄)² )the square is what lets one far-out row dominate

These percentages hold for a normal distribution and only then

Interquartile Range (IQR)Q3 − Q1, the width of the middle half: it does not care how far the outlier sits

IQR = Q₃ − Q₁Q₁, Q₃ are the 25th and 75th percentiles

In this example, moving the largest value farther out leaves the quartiles unchanged

Control Limits and the Normal CurveA normal curve puts about 0.27% of its mass outside μ ± 3σ

0.27% assumes a normal distribution with known parameters; estimated limits can behave differently

Case study: Heavy Tails in Price ChangesMandelbrot, 1963: the risk models of the day assumed Gaussian moves

Quieter than Gaussian on most days, and far wilder on the rare ones

Frequency Distribution

Box PlotFences flag extreme values; whiskers end at observed values inside them

IQR = Q₃ − Q₁|fences at Q₁ − 1.5·IQR and Q₃ + 1.5·IQR

Fences are thresholds; whiskers end at data points, and flagged values still need investigation

Histogram as Empirical DistributionBins summarize how many observations fall in each interval

Bin counts show distribution shape while hiding within-bin detail

Demo: Histogram Bin CountSame 420 numbers, bin count is the only thing changing

1. Compare 4, 24 and 70 bins.   2. Which pattern survives your choices?

Bins 24 Reset

Answer: Histogram ChoicesRead what changed before choosing a setting

Setting	Interpretation
Few bins	Distinct peaks can merge
Many bins	Sparse bars can mimic structure
Several settings	Report stable distribution features

The setting changes the picture; the underlying data stay fixed

Visual Exploration

Recap: Five Standard ChartsThe five you already know

Every one of these needs you to already know which columns matter

Data Manifold

Data ManifoldA lower-dimensional structure can explain variation in many measured columns

These two columns vary mostly along one direction

Clustering with k-meansIt always returns groups: the same points after asking k-means for three

Choosing k asks for that many clusters; it does not establish the true group count

Demo: k-means ClusteringSame 150 points every time, k is the only input k-means gets

1. Compare k = 2, 3 and 6.   2. Can the smallest WCSS select k?

k 3 Reset

Answer: Cluster CountRead what changed before choosing a setting

Setting	Interpretation
k = 2	Groups merge
k = 3	Three groups in this synthetic example
k = 6	Groups split; WCSS can still fall

The setting changes the picture; the underlying data stay fixed

Case study: Marketing SegmentationA cluster needs a name to be useful: from a cluster number to a readable persona

A cluster number is useless until someone can say what makes that group different

Dimensionality Reduction

Principal Component Analysis (PCA)Rotate the axes so the spread lands on the first few

The three arrows are the new axes. The longest one points along the widest spread

Projection onto a ComponentPCA turns each original row into coordinates along the fitted directions

Two measured columns

temperature · vibration

2 numbers per observation

→

Fitted PCA direction

center on training mean
project onto the learned axis

→

One component

z = (x − μ) · v

1 number per observation

Keeping fewer coordinates discards variation along the omitted directions

Choosing the Number of ComponentsRetained variance measures preserved spread; useful information depends on the task

Four components keep 91% of variance; task-relevant information still needs checking

t-SNE vs PCAt-distributed stochastic neighbor embedding separates what PCA leaves overlapping

The same digits, twice: the classes overlap on the left and separate on the right

Demo: t-SNE PerplexitySame two datasets every time, only the perplexity moves

1. Compare perplexity 2 and 30.   2. Can a visible island prove a real cluster?

Perplexity 30 Reset

Answer: Reading t-SNERead what changed before choosing a setting

Setting	Interpretation
Small perplexity	Local islands appear, even in random data
Larger perplexity	Some islands disappear
Original data	Check neighbors, stability and domain meaning

The setting changes the picture; the underlying data stay fixed

Reading a t-SNE PlotCheck stability; axes, blob areas, and gaps do not directly measure the original geometry

• The axes carry no meaning. PCA rotates, so its axes can be read; these two cannot
• Blob area does not directly measure density. Dense regions are inflated and sparse ones are shrunk
• The distance between two clusters is not their dissimilarity. Local neighborhoods are an approximate optimization target

Time Series

Time Series and StationarityOrder, trend, and seasonality contain information that one mean cannot capture

A changing mean violates weak stationarity; a single average hides the trend

Time Domain vs Frequency DomainThe same signal: a tangled wave in time becomes two bars in frequency

X(f) = ∫ x(t) e^−2πift dtthe Fourier transform; the fast Fourier transform (FFT) computes it quickly

What looks tangled in time becomes two bars in frequency

Statistical Exploration

Two Variables

Scatter PlotOne point per observation shows the relationship before a fitted model

Direction, shape, spread, and the rows that sit apart, all before anything is computed

Demo: Anscombe’s QuartetSame mean, same variance, same correlation, same fitted line

1. Do equal summaries imply equal shapes?   2. Which plot makes the fitted line misleading?

I II III IV All four

Answer: Anscombe’s QuartetThe same summary statistics describe different relationships

What stays the same	What the plots reveal
Mean and spread	Different geometry
Correlation ≈ 0.82	Curve, outlier and leverage
Line y ≈ 3 + 0.50x	A line is not equally informative in every plot

Plot the observations before interpreting a fitted relationship

Fitting a LineA fitted line summarizes a linear trend; residuals show departures

A fitted line defines residuals; a distant point warrants investigation

Covariance of Two ColumnsTwo columns can spread out together, measured in the units of both

cov(x, y) = (1/(n−1)) ∑_i (x_i − x̄)(y_i − ȳ)the SD formula with the second factor taken from another column

Positive products add, negative products subtract; their weighted sum determines the covariance

Pearson Correlationr measures a straight line, not any relation: a scatter plot reduced to one number

r = ∑_i(x_i−x̄)(y_i−ȳ) ⁄ √( ∑_i(x_i−x̄)² ∑_i(y_i−ȳ)² )the same thing as cov(x, y) / σ_xσ_y

Near-zero r rules out strong linear association; nonlinear relationships may remain

Spearman’s Rank CorrelationRanks catch the curve that Pearson underrates: does it keep going one way?

ρ_s = corr(rank(x), rank(y))

Rank correlation captures the monotone curve: Pearson 0.76, Spearman 0.92

Many Variables

Correlation MatrixPairwise correlations reveal related groups and a weakly related humidity column

Speed, vibration and temperature are strongly related; humidity stays weakly related

Matrix PlotReorder both axes by species and similar samples form blocks along the diagonal

Reordering both axes reveals similarity blocks that a random order hides

Correlation to Cause

What wrote this table?Exploration reads the rows; what the rows mean is decided one level up

The correlation is observed in the table; its meaning depends on the process above it

Causal GraphsThe same correlation can reflect either causal direction or a shared cause

Three structures give one correlation; the correlation alone cannot tell them apart

Takeaway: Look before you summarizeExploration reveals patterns whose meaning depends on measurement and modeling choices