IMEN891K / IDSC501 / AXGS-501 · Data Engineering · Fall 2026
5. Pre-processing II: Transformation & Reduction
Sangwoo Mo · POSTECH
Industrial & Management Engineering
Industrial & Management Engineering
Where We Are
Recap: Four Stages of Preprocessing
Transformation
Recap: Data TypesNumbers get rescaled; discretization turns numbers into categories
The type of a column decides the tool
Three Branches of Transformation
Scaling Numbers
Why Scale at AllDistance-based models add up raw units, so the widest column decides
In raw Euclidean distance, the rpm difference dominates the cycle-time difference
The Evaluation Must Match DeploymentDefine what will be new at prediction time before splitting the records
| Deployment question | Useful split | What a random row split can hide |
|---|---|---|
| Predict future events | Earlier train; later test | Future information reaches training |
| Predict new machines | Hold out whole machines | Same-machine patterns occur on both sides |
| Predict new customers | Hold out whole customers | Repeated identity makes the test easier |
A held-out row can still share an entity or future information with training
Caveat: One Machine on Both Sides
Random rows
| Machine | Train | Test |
|---|---|---|
| M1 | 80 sampled cycles | 20 held-out cycles |
| M2 | 80 sampled cycles | 20 held-out cycles |
New-machine evaluation
| Machine | Role |
|---|---|
| M1 | Train |
| M2 | Test |
Testing new machines requires machines that training has never seen
Fitted Transformation StateFit learns parameters; transform reuses the saved parameters
| Step | Fit on train | Transform new rows |
|---|---|---|
| Scaling | min, max or mean, SD | reuse the same numbers |
| Encoding | category vocabulary | reuse the same columns |
| PCA | center and axes | project onto the same axes |
Fit once on training rows, then apply the saved state
Linear ScalersThree scalers, one shape: subtract a center, divide by a spread
Picking a scaler is picking how many bad rows it takes to move the answer
Min-Max Scalingx′ = (x − min) / (max − min) maps the training range onto [0, 1]
The minimum maps to 0, the maximum to 1. Fit both on train only
Demo: Fit on Train, Apply to TestTrain cycle times stay at 28, 32 and 36 seconds
1. Can a transformed test value exceed 1? 2. Does refitting change the training rows?
40 s
Answer: Reusing a ScalerAt test = 40, the saved training range produces 1.5
| Fitting rows | min / max | train 36 → | test 40 → |
|---|---|---|---|
| Train only | 28 / 36 | 1.000 | 1.500 |
| Train + test | 28 / 40 | 0.667 | 1.000 |
New observations can lie outside the fitted training range
Caveat: Min-Max and OutliersOne outlier is enough to squash every other value
The maximum defines the range, so the maximum defines everything
Z-Score StandardizationEach training column gets mean 0 and standard deviation 1
z = (x − μ) / σthe shape is untouched; only the axis is relabeled
A right tail stays a right tail. Standardizing does not make a column normal
Robust ScalingThe median and the IQR barely move when one row goes wrong
x′ = (x − median) / IQRIQR = interquartile range
The median moves to zero and the outliers stay far out. That is intended
Demo: Feature Scalers300 cycles from one computer numerical control (CNC) machine, both columns on one axis
1. Compare scalers with a mistyped rpm. 2. Did scaling repair the error or change the reference statistics?
Answer: Scaling and OutliersScaling changes the reference statistics, not the recorded event
| Scaler | Reference statistics | Effect of an extreme value |
|---|---|---|
| MinMax | min and max | clean values compress into a narrow range |
| Standard | mean and SD | center and spread can move |
| Robust | median and IQR | resists some extremes; keeps the outlier |
Scaling changes representation; it does not validate or remove a value
Log TransformLog turns ratios into equal steps
log2(2x) − log2(x) = 1x > 0
Each doubling becomes the same gap. Normality is not guaranteed
Unit Vector NormalizationScale each row, not each column, to length 1
x′ = x / ‖x‖2then x′Ty′ is exactly the cosine similarity
Every row lands on the unit circle; the dot product is now cosine similarity
Encoding and Binning
Encoding vs BinningOne turns categories into numbers, the other turns numbers into categories
What the model can use decides which way you cross
Nominal and Ordinal CategoriesA numeric code can introduce order and spacing into the model
Nominal: no known order
| Color | Arbitrary code |
|---|---|
| red | 1 |
| green | 2 |
| blue | 3 |
Ordinal: a known order
| Size | Ordered code |
|---|---|
| small | 1 |
| medium | 2 |
| large | 3 |
Color codes invent an order; size codes express an existing order
One-Hot EncodingOne indicator column per category, so no order is invented
x → ( 𝕀[x = c1], …, 𝕀[x = ck] )k categories become k columns whose sum is always 1
With an intercept, k one-hot columns are linearly dependent. Dropping one removes this dependency
Choosing an EncodingOrder, the number of categories and the model decide the encoding
| Data meaning | Starting choice | Check |
|---|---|---|
| Unordered categories | One-hot | Width and unknown values |
| Known ordered levels | Ordinal codes | The supplied order matches the meaning |
| Text or many related categories | Learned embedding | Vocabulary, model version and validation |
Start with the category meaning, then check width and held-out behavior
Demo: Categorical EncodersSix training rows; reuse their category vocabulary for test
1. Compare one-hot with integer codes. 2. Apply the fitted vocabulary to an unseen test color.
BEFORE
AFTER ·
Answer: An Unseen CategoryA fitted encoder needs an explicit policy for a new value
| Encoder policy | Blue at test time | Consequence |
|---|---|---|
| One-hot, ignore unknown | [0, 0, 0, 0] | keeps four fitted columns |
| Integer codes, unknown = -1 | -1 | unknown remains distinct |
| Add a test-only Blue column | five columns | changes the fitted representation |
Keep the fitted vocabulary; define how unknown values are represented
Vector EmbeddingA learned model maps each category to a short real vector
Distances now differ between pairs. In one-hot, every pair of categories is equally far apart
An Encoding Cannot Restore a Binned ValueBinning loses within-bin differences; encoding represents the resulting category
Measured number
cycle time = 28.4 s
→
Bin category
B2: [25 s, 30 s)
→
One-hot representation
B1 B2 B3 0 1 0
A numeric output does not imply that the original measurement was preserved
Why DiscretizeContinuous values give almost every row its own case
Four bins replace 29 values. Where the cuts go is the whole question
Binning by HistogramEqual width keeps the scale, equal frequency keeps the counts
width = (max − min) / kvscount = n / kthe same k, two different things held constant
Equal width: 26, 8, 1, 2, 3 rows. Equal frequency: 8 in every bin
Making Features
Hand-Crafted FeaturesNo column holds it, so you compute one: TF-IDF (term frequency × inverse document frequency)
| Term in one maintenance log | tf | df of 1,000 logs | log(N/df) | Weight |
|---|---|---|---|---|
| the | 12 | 1,000 | 0.00 | 0.0 |
| pump | 4 | 300 | 1.20 | 4.8 |
| cavitation | 2 | 12 | 4.42 | 8.8 |
w = tf · log(N / df)N = 1,000 logs · df = how many of them contain the term
The most frequent word scores zero. A person decided that is the right behavior
Learned FeaturesThe network learns the features and the classifier together
Fit your own encoder or reuse a versioned pretrained encoder
Case study: YouTube-8MThe dataset ships features, not videos
Feature extraction is also compression, and it is irreversible
Reduction
Three Branches of Reduction
Why Reduce at AllColumns, rows and bytes each cost something different
Removing rows or columns changes learning; lossless compression preserves the data
Fewer Columns
Extraction or SelectionKeep some original columns, or mix all of them into new ones
Selection keeps names you can read. Extraction keeps information you cannot name
Recap: PCA as a Fitted TransformReuse the training center and axes when new observations arrive
znew = (xnew − μtrain) Vtrain,k
Projection keeps selected directions and discards the rest
Demo: PCA Projection AngleTurn the axis and watch how much of the variation survives
1. Find a direction that keeps more variance. 2. Is retained variance enough to choose model features?
Answer: What PCA KeepsLarge retained variance does not guarantee useful predictive features
| Question | Check |
|---|---|
| What survives? | Variation along the chosen training axes |
| What disappears? | Variation in discarded directions |
| Does the model still work? | Evaluate with held-out observations |
Validate the retained representation, not only its variance
Three Families of Feature SelectionCompare model use and relative fitting cost within one selection procedure
| Family | What it looks at | Model fits | Blind spot |
|---|---|---|---|
| Filter | the columns alone, no model | none | a pair that only works together |
| Wrapper | the model’s score on a subset | many | the search itself overfits |
| Embedded | the model’s own coefficients | one | the answer is tied to that model |
Compare cost, interactions and held-out performance before choosing a selection method
Wrapper Feature SelectionA wrapper compares feature subsets through repeated model fits
Candidate feature subset
temperature + vibration
→
Fit and validate
Train on training folds Score on validation folds
→
Compare another subset
temperature + current Repeat within the same protocol
Repeated subset search can overfit the validation evidence
Caveat: Features That Work Only TogetherA univariate score can miss a predictive feature interaction
Four synthetic observations
| valve A | valve B | defect |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
One column alone
A = 0: one pass, one defect
A = 1: one pass, one defect
A alone cannot separate them
The pair together
A ≠ B → defect
A = B → pass
Useful combinations can look uninformative when each column is scored alone
Embedded MethodsSome models select features during fitting; the result depends on the model and training data
| Feature | Illustrative fitted coefficient | Interpretation |
|---|---|---|
| Vibration | 0.8 | Used by this fitted model |
| Temperature | 0.3 | Used by this fitted model |
| Humidity | 0.0 | Removed by this fitted model |
A zero coefficient describes one fitted model, not the absence of a real relationship
Fewer Rows
Sampling vs CurationBoth drop rows: sampling follows a design, curation follows quality
| Sampling | Curation | |
|---|---|---|
| Goal | A smaller set with known proportions | A better set: quality, consistency, coverage |
| Techniques | Random, systematic, stratified, cluster | Filtering for quality and coverage |
| Used for | Less computation, class imbalance | Training sets for large models, labeling budgets |
| Proportions | Kept, or changed on purpose (imbalance) | Changed by what the filters remove |
Sampling decides which rows to keep; curation decides how good they must be
Sampling DesignThe selection rule determines which parts of the population survive
| Method | How rows are chosen | Failure to check |
|---|---|---|
| Simple random | Draw rows at random | Small groups may be missed |
| Systematic | Every kth record | A periodic list can align with k |
| Stratified | Sample within each group | Keep the intended group proportions |
| Cluster | Select whole groups | One factory may not represent the rest |
Check the population structure before choosing the sampling rule
Sampling to Fix ImbalanceDropping majority rows raises the minority share from 17% to 44%
Resampling changes class prevalence. Evaluate and calibrate on the deployment distribution
Caveat: Survivorship BiasThe rows you kept are not the rows you lost
This shows the bombers that came back. Reinforce where there are no dots
Data CurationCuration filters for quality instead of sampling at random
This pipeline keeps 1.4% of the crawl as the training set
Fewer Bytes
Lossless or LossyCompression restores the original bit for bit, or only an approximation
The lossy box is smaller, and its changes stay
Lossless Master CopyKeep one lossless copy, because a lossy decision cannot be revisited
| Lossless | Lossy | |
|---|---|---|
| Restores | The original, bit for bit | An approximation |
| Typical data | Text, code, logs, tabular | Images, audio, video |
| Formats | ZIP, PNG, GZIP, Parquet with Snappy or Zstd | JPEG, MP3; lossy H.264 / WebP |
| Ratio | Limited by entropy | Often much higher, tunable |
| The catch | Sometimes not small enough | The decision cannot be revisited |
Text and logs stay lossless; media is where lossy pays
Fitting Preprocessing StepsFit on train, apply to test: every fitted step remembers something from the rows it saw
| Step | What it remembers | What leaks if you fit on everything |
|---|---|---|
| Imputation | the column median | test rows move the median |
| Scaling | mean and standard deviation | the test spread enters every train row |
| Encoding | the list of categories | a category that exists only in test |
| Binning | the bin edges | edges drawn around test values |
| Feature selection | the columns kept | columns kept because they help on test |
| PCA | the components | axes rotated to fit test variance |
z = (x − μtrain) / σtrainthe same two numbers go to test and to every future row
Refit inside every fold; ship the final fit to testing and serving