IMEN891K / IDSC501 / AXGS-501 · Data Engineering · Fall 2026

Midterm Review

Sangwoo Mo · POSTECH
Industrial & Management Engineering

Outline: Midterm Review

Proposal Review

Midterm Review

By Course Code

01 · Proposal Review

Proposal Review

Midterm Review

By Course Code

How Scores Were Processed

Peer Scores

Peer Scores by CriterionFinal criterion scores add up to the total

Who Rated WellThe top three raters get bonus points

Team 03: Conference contacts

AI feedback (slides only)

What worked

· PostgreSQL design: 16 entities with keys, provenance and sync metadata

What was missing

· Ingestion examples only; no loaded counts or executed query results

Team 06: Spot welding

AI feedback (slides only)

What worked

· Seven data issues diagnosed with numbers; 11,939 rows cut to 2,184 clean rows

What was missing

· Modeling columns defined, but no database engine or store chosen

Team 08: Recall monitoring

AI feedback (slides only)

What worked

· 44,518 recall records from three APIs loaded into SQLite with four tables

What was missing

· Seven problem-framing pages before the built pipeline and results

Team 10: Parking rules

AI feedback (slides only)

What worked

· Six public datasets joined to one row per building, 86,987 buildings, all counted

What was missing

· Joined rows shown, but no database engine or primary keys

Team 15: Chest X-ray tail

AI feedback (slides only)

What worked

· Dataset profiled with numbers: 68,058 train images, 7 head classes hold 93.4%

What was missing

· Synthetic generation method unnamed; no check that generated images are valid

Team 05: Hydraulic pump

AI feedback (slides only)

What worked

· Two-store design with candidates, selection criteria and four save checks

What was missing

· No loaded sensor counts or results yet; demo still in preparation

Team 11: Meal planning

AI feedback (slides only)

What worked

· Four stores designed: recipe, ingredient, rules in SQL; user profile as document

What was missing

· Sample records only; collection volume and coverage not reported

Team 04: Student funding

AI feedback (slides only)

What worked

· One real notice already parsed into the JSONB schema with stored values

What was missing

· Data so far is one program, three cases, five documents; no crawl count

Team 01: Highway icing

AI feedback (slides only)

What worked

· Seven sources with cadence, three held-out sensors, baseline and leakage-safe CV

What was missing

· Wide scope: terrain model, risk prediction, live map and LLM agents

Team 09: Neonatal EEG

AI feedback (slides only)

What worked

· Concrete public dataset: 79 neonates, 19 channels, 256 Hz, three annotators

What was missing

· No model trained or selection performance measured yet

Team 14: 3D scanning

AI feedback (slides only)

What worked

· Real EDA already run: coverage versus views for bottle and dragon

What was missing

· Object dataset name and total size not specified

Team 12: Meme coins

AI feedback (slides only)

What worked

· 14,773 daily price rows for 10 coins already crawled from CoinMarketCap

What was missing

· Social data not collected yet; Reddit access still pending

Team 16: Lab finder

AI feedback (slides only)

What worked

· Clear schema and raw-to-clean normalization rules with real DI page examples

What was missing

· Seven sample rows shown; full crawl coverage and totals unreported

Team 17: Grocery restrictions

AI feedback (slides only)

What worked

· 6.2 million products collected from seven official APIs and open datasets

What was missing

· Dense slides and wide scope: meal plans, chat, OCR on top of catalog

Team 13: Experience paths

AI feedback (slides only)

What worked

· Clear worked example: filter-delay case shows what the system returns

What was missing

· No papers fetched or experience records counted yet

Team 02: Disclosure risk

AI feedback (slides only)

What worked

· Real OpenDART filings counted: 1,512 rights issues in one year

What was missing

· Stores named, but no table definitions or keys shown

Team 07: Port analytics

AI feedback (slides only)

What worked

· Clear pain point: BPA portal serves fixed tables needing domain knowledge

What was missing

· Too abstract: agent workflow only, no port data loaded or schema

02 · Midterm Review

Proposal Review

Midterm Review

By Course Code

Score Distribution

Item Scores

Q1 Sensor as Encoder

A temperature sensor acts as an encoder: it converts a physical signal into a form that can be recorded digitally.

True ✓

False

Why. An instrument turns an event in the world into a record; a sensor encodes temperature as numbers

Q2 Correlation and Cause

Production volume and defect rate are strongly correlated. This proves that lowering production will lower the defect rate.

True

False ✓

Why. Correlation cannot say what happens under an intervention; a shared cause, such as rushed overtime, can drive both

Q3 Missing Values Before Filling

Some customer records have missing values. The team should fill them all before checking why they are missing.

True

False ✓

Why. Why a value is missing decides how to treat it; missingness itself can carry information

Q4 Schema Alignment vs Entity Matching

Two systems use different customer IDs. Giving their columns the same names is enough to match each customer.

True

False ✓

Why. Same column names align the schema; deciding which records are the same customer is entity matching, a separate step

Q5 Scaling Before Distance

A model measures customer similarity using age and income. Scaling helps stop income’s larger numbers from dominating the distance.

True ✓

False

Why. Distance adds squared differences; income in millions swamps age in tens. Scaling puts both on one footing

Q6 One-Hot and Order

A model must distinguish low, medium and high priority. One-hot encoding tells the model which level is higher.

True

False ✓

Why. One-hot gives each level its own column with no order between them; ordered levels need ordinal encoding

Q7 Selection vs Extraction

A team wants fewer inputs while keeping each variable’s meaning. Feature selection does this by combining variables into new ones.

True

False ✓

Why. Selection keeps a subset of the original variables, so each keeps its meaning; combining them is extraction, such as PCA

Common slip: both reduce inputs, only extraction makes new ones

Q8 Keys Between Tables

A relational database can store each customer’s details once. Orders can refer to that customer using a key.

True ✓

False

Why. Customers live once in their own table; each order carries the customer key, a foreign key

Common slip: copying customer details into every order

Q9 Flexible Documents

Some customer profiles have fields that others do not. A document store can keep these profiles in the same collection.

True ✓

False

Why. A collection does not force one schema; documents with extra fields sit side by side

Common slip: treating a collection like a table with fixed columns

Q10 Results as Inputs

A SQL query’s result can be a table in another query. A MongoDB aggregation stage can pass its documents to the next stage.

True ✓

False

Why. Both compose: a SQL result is a table another query reads; each stage hands documents to the next

Q11 API or Crawling

You need data from two sites: one has an API; the other only has web pages. Which methods fit the first and second sites?

AWeb crawling; API calls

BAPI calls; web crawling ✓

CAPI calls; sensor measurement

DSynthetic data; web crawling

Why. Use the route the source offers: an API returns structured records; pages without an API need crawling

Q12 Typical Value Under Outliers

A few houses have prices far above the rest. Which measure of a typical price is less affected by these extreme values?

AMean

BRange

CStandard deviation

DMedian ✓

Why. The median depends on order, not size: a few huge prices move the mean, not the middle value

Q13 Same Statistics, Different Shapes

Two stores have the same means, variances and correlation for sales and ad spending. Why still compare their scatter plots?

AMatching statistics mean the data have the same shape

BMatching statistics mean the data have the same outliers

CDifferent patterns can produce the same statistics ✓

DA scatter plot cannot show anything beyond the correlation

Why. Anscombe’s quartet: four datasets share mean, variance and correlation yet look completely different

Q14 Smoothing a Noisy Trend

Sensor readings fluctuate rapidly because of noise. You want a smoother trend on a dashboard. Which method fits?

AMoving average ✓

BOne-hot encoding

CkNN imputation

DZ-score standardization

Why. A moving average replaces each point with the mean of its neighbors, damping short-term noise

Q15 Sampling Every Factory

You need a sample that includes every factory. Some factories have far more records than others. What should you do?

ATake the first records in the combined dataset

BSample only from the largest factory

CRandomly sample at least one record from every factory ✓

DRemove unusually large or small values

Why. Group by factory and draw at least one record from each, so no factory is left out

Q16 SELECT-FROM-WHERE

This query builds a product list. What does the list contain?

SELECT name
FROM Product
WHERE category = 'Electronics';

ANames of Electronics products ✓

BAll fields of Electronics products

CNames of all products

DThe number of Electronics products

Why. SELECT picks the name column; WHERE keeps only Electronics rows

Q17 Basic JOIN

Every product has a matching company. This query builds a supplier report. What does it show?

SELECT p.name, c.country
FROM Product p JOIN Company c
  ON p.company_id = c.company_id;

AThe number of products in each country

BEvery possible product–company pair

CEach product and its category

DEach product and the country of its company ✓

Why. ON pairs each product with the company whose ID matches; company_id is unique, so one row per product

Q18 GROUP BY and Average

A dashboard runs this query. What does each result row describe?

SELECT category, AVG(price) AS average_price
FROM Product
GROUP BY category;

AEach product and its price

BOne average price for all products

CEach category and its average product price ✓

DEach company and its average product price

Why. GROUP BY category makes one row per category; AVG(price) is computed inside each group

Q19 Access Pattern to NoSQL Type

A service must (i) find a session by its token and (ii) follow links between friends. Which types fit, in that order?

AGraph; key-value

BKey-value; graph ✓

CDocument; wide-column

DWide-column; document

Why. A token lookup is a get by key: key-value. Following friend links is traversal: graph

Q20 Aggregation Stages

Each sale records a city, product category and amount. You need total Electronics sales in each city. Which steps?

AKeep Electronics; group by category; sum amounts

BKeep Electronics; group by city; count the sales

CGroup all sales by city; sum amounts; sort totals

DKeep Electronics; group by city; sum amounts ✓

Why. Filter first ($match), then group by city ($group), then sum the amounts ($sum)

Common slip: grouping by category gives one total, not one per city

03 · By Course Code

Proposal Review

Midterm Review

By Course Code

Midterm by Course CodeAXGS-501 averages 2 points lower, from a longer low tail

Proposal by Course CodeTeam scores are shared, and the three codes sit within half a point

Correct Rate by Lecture BlockThe AXGS-501 gap grows from Part I to SQL and NoSQL

Thank you for your hard work
on the proposal and midterm!