IMEN891K / IDSC501 / AXGS-501 · Data Engineering · Fall 2026

Review Session

Sangwoo Mo · POSTECH
Industrial & Management Engineering

Outline: Review Session

01

Collection and Exploration

Lec 2–3 · records, labels and patterns

02

Data Preparation

Lec 4–5 · changes to values and representation

03

SQL

Lec 6–8 · read, match and summarize rows

04

NoSQL

Lec 9–10 · models, documents and updates

01 · Collection and Exploration

01

Collection and Exploration

Lec 2–3 · records, labels and patterns

02

Data Preparation

Lec 4–5 · changes to values and representation

03

SQL

Lec 6–8 · read, match and summarize rows

04

NoSQL

Lec 9–10 · models, documents and updates

Outline: Data Collection

01Define the RecordWhat does one row hold?one row = one part02Build the Collection PathWhere does each value come from?reuse, observe, generate03Collect Reliable LabelsHow do we get reliable labels?annotate, observe, validatewhat comes back changes what you collect next

Online and Offline Data

Where is the event recorded?Onlinedigital assets and logsAPI callsa system provides an interfaceWeb crawlingonly a webpage is availableOfflinephysical observationsSensormeasure a physical signalHuman recordwrite down an observationA digital record the system can store and check
Locate the observation source before choosing the collection method

API or CrawlerPrefer an explicit interface; page extraction also depends on the page layout

API their system a contractversioned, documented your rows An explicit interface can be versioned and checked CRAWLER their system a page for humansredesigned on their schedule your selector 0 rows no error raised A redesign breaks extraction without an HTTP error
An explicit interface can be checked; page extraction also depends on layout

Sensors as Encoders

Physical signalEncoderRecorded representationTemperatureTemperature sensorencoder09:14:02 41.8 °C09:14:03 41.9 °C09:14:04 42.1 °CVibrationVibration sensorencodert = 0.000 s 0.12t = 0.001 s −0.08t = 0.002 s 0.15
Measurement turns a physical event into a recorded observation

Record Meaning and Context

38.6one recorded valueEntityPart P-104Event time09:14:02Measurement and unitcycle time, secondsSource and definitionCNC-02 log; defect rule v3
A value becomes usable when its meaning and context are recorded

Generating Training Examples

Annotationadd a label to an existing inputinputdefectCheckDefinition and label qualityAugmentationchange an input, keep its labelinputrotateddefectdefectCheckWhether the label still appliesSynthesisgenerate a new inputgeneratornew inputCheckRealism, coverage and usefulness
Generated inputs and candidate labels still require validation

Sources of LabelsA label can come from deliberate annotation or from an observed outcome

SourceExampleMain collection problem
Human annotationAn inspector judges a partMake the guideline reproducible
Natural labelA delivered shipment has an arrival timeRecord and link the outcome

Model-generated labels remain candidates until checked

Both sources need a definition and a quality check

Checking Label Quality

DefinitionWhich event is “defect”?defecttolerance violationreturncomplaintAgreementSame guideline, same answer?part 17R1“defect”R2“pass”disagree → refine the guidelineIndependent auditWhat does the selection miss?chosen by policyrandom audit
A label needs a clear definition, reproducible judgment and independent checks

Missing OutcomesKeep pending, unobserved and negative outcomes distinct

decidedrow written at 09:14:02pendingsent to inspectionnot inspectedshipped without a lookobserved: passlabel = 0observed: defectlabel = 1still missingno label after 30 daysresult arrivesnever this arrownot inspected ≠ passA model trained on "label or 0" learns that skipped parts are good
Not inspected is different from an observed pass

Selected OutcomesThe inspection policy controls which outcomes become labels

100 produced parts

20 selected for inspection
Labels will be observed

80 shipped without inspection

Outcomes unknown
They are not recorded as pass

Random audit of 5

25 labels in total
75 outcomes remain unobserved

Audit outside the model-selected sample to observe what it missed

Collection Boundaries

Purpose and accesswhy it is neededwho may use it1Rights and confidentialitymay we collect,reuse and transfer?2Protection and provenancerestrict access; keepsource, time, definition3Collect or reusedecision recordedEach gate is a recorded decision before any row is collected
Collect only what the purpose needs, with permission and protection

Outline: Data Exploration

01 Descriptive Analysis Reduce the data to a few numbers 02 Visual Exploration Find patterns by plotting the data 03 Statistical Exploration Read relationships between variables

Recap: Mean, Median, ModeThree answers to “the typical value”

mode 3median 4.5mean 5012345678910sorted: 1, 3, 3, 6, 8, 9
From 9, 3, 1, 8, 3, 6: mean 5, median 4.5, mode 3

Extreme Values and the Median

1, 3, 3, 6, 8, 9020406080100mean 5median 4.51, 3, 3, 6, 8, 99020406080100mean 20median 4.5The mean uses every value; the median uses the middle positions
An extreme value pulls the mean; the median is less sensitive

Variability and Robust Summaries

-20020406080100Without the extreme valuemean ± SD: 1.8 to 8.2median 4.5, IQR 3 to 7.5-20020406080100With the extreme valuemean ± SD: -18.8 to 58.8median 4.5, IQR 3 to 7.5
Report spread as well as center, and inspect the tails

Histogram as Empirical DistributionBins summarize how many observations fall in each interval

Histogram
Bin counts show distribution shape while hiding within-bin detail

Clustering with k-meansIt always returns groups: the same points after asking k-means for three

The same points assigned to three clusters
Choosing k asks for that many clusters; it does not establish the true group count

PCA and t-SNE

PCAt-SNEPC1PC2distance between clusters?Shows: directions retaining large varianceNot: the most predictive or causal directionsShows: local neighborhoodsNot: reliable distances between clusters
A projection is a view of the data, with method-specific limits

Time Series Views

Time domainWhen did the level, trend or variation change?level shiftFrequency domainWhich repeating cycles contribute?daily cycleAcross time periodsIs the process stable enough to compare?period Aperiod BTime plots compare values over time; scatter plots compare two variables
Preserve time order and check whether the process changes

Anscombe’s QuartetSimilar summary statistics, different scatter plots

The four Anscombe plots revisited from Lecture 3
The same summaries can hide different relationships

Pearson and Spearman Correlation

Linear relationPearson r = 0.97Spearman ρ = 0.96Monotone curvePearson r = 0.80Spearman ρ = 1.00Pearson: linear association; a nonlinear relation can score lowerSpearman: rank association; neither one shows causation
Choose a relationship summary, then check the scatter plot

Causal GraphsThe same correlation can reflect either causal direction or a shared cause

ABA causes BABB causes AABZZ causes both
Three structures give one correlation; the correlation alone cannot tell them apart

02 · Data Preparation

01

Collection and Exploration

Lec 2–3 · records, labels and patterns

02

Data Preparation

Lec 4–5 · changes to values and representation

03

SQL

Lec 6–8 · read, match and summarize rows

04

NoSQL

Lec 9–10 · models, documents and updates

Outline: Four Stages of Preprocessing

01 Cleaning Wrong values in one table 02 Integration one view Tables that disagree 03 Transformation 12.4 0.42 0.03 0.00 870 1.00 Values a model can read 04 Reduction Less data, same signal

Data Imputation

Imputation estimates a missing value using observed data.

machinetemppressuretargetM1712.120M2742.324M3732.2?Keep missing or remove the rowDoes absence mean something? Which rows are lost?Fill from a column summaryIs a typical value a useful estimate here?Predict from related observationsDo temp and pressure carry information?
An imputed value is an estimate, not a new observation

Observed and Filled ValuesUse other observed columns to find comparable rows

Observed measurements

parthelpertarget
P11020
P21224
P311NULL

Estimate for P3

Neighbors in helper: P1 and P2
(20 + 24) / 2 = 22

Stored with provenance

parttargetestimated
P322Yes
The filled number is an estimate; keep its origin distinguishable

Caveat: Mean ImputationEvery hole gets the same value, and the spread of the column collapses

Original distribution versus zero imputation and mean imputation
Mean stays 2.08; standard deviation falls to 1.76 and a spike appears

Caveat: Why is it missing?

Holes independent of the value 25 × 26 121 29 91 30 × 30 110 31 × 44 118 46 93 48 × 51 × 52 116 54 × age IQ
Holes concentrated in the younger half 25 × 26 × 29 × 30 × 30 × 31 × 44 118 46 93 48 116 51 141 52 97 54 104 age IQ

Below, no young IQ was ever observed. No imputer fixes this. The question is not which method to use but why those rows are missing, and that answer lives in how the data was collected

Noise vs OutliersSmooth measurement noise; investigate outliers before changing them

Noise: every value is a little wrong smooth all 1,200 rows survive Outliers: a few values sit far away if confirmed: remove three rows are gone
Smoothing changes values; removal drops rows

Moving Averages for StreamsSimple (SMA) and exponential (EMA) moving averages use only current and past samples

alarm limit 66 °C EMA, α = 0.15 SMA, w = 20 4 samples of alarm delay fault starts here
Both use past samples; here EMA crosses four samples before SMA

Consistency Checks

A time interval cannot end before it startsneeds investigation09:0009:2009:4010:00start 09:40 → end 09:10A value must use the declared unit and typeconflicts with its definitioncycle time (s)38.641.20.64 minA reference must identify an existing entityincomplete or wrong linkorder 5521customer C-999C-999not found
A failed check identifies a problem, not its correct replacement

One Customer, Three Systems

CRMHanil Precision Co., Ltd.C-10482SCMHANIL PRECISIONV0000734ERP한일정밀(주)2010-HPsamecustomer?Different IDs and spellingsneed evidence of identity
Different source IDs and spellings do not establish a shared identity

Three Questions of Integration

Schema mappingWhich columns mean the same?cust_namecustomerRenaming columns enough?Sometimes, if units and definitions matchEntity matchingWhich rows are the same customer?C-10482V0000734Renaming columns enough?No: different IDs need evidenceValue fusionWhich value survives a conflict?054-279-0001054-279-1234Renaming columns enough?No: choose a source or rule
Matching columns and matching entities solve different problems

Resolving Conflicting Values

CRM: 054-279-0001updated 2024ERP: 054-279-1234updated 2026Rulenewer contact replaces older054-279-1234kept: source ERP, newerTrust a source for a fieldFinance for recognized revenueCompare event time and meaninga newer current numberKeep the evidencesource IDs and the reason
Entity matching and value reconciliation are separate decisions

Recap: Four Stages of Preprocessing

01 Cleaning Wrong values in one table 02 Integration one view Tables that disagree 03 Transformation 12.4 0.42 0.03 0.00 870 1.00 Values a model can read 04 Reduction Less data, same signal

Why Scale at AllDistance-based models add up raw units, so the widest column decides

two columns from one machine, on one raw axis 0 500 1,000 1,500 2,000 2,500 3,000 3,500 spindle speed (rpm, revolutions per minute): 1,679 to 3,009 cycle time: 28 to 36 s, three pixels wide how far apart two rows can be, column by column Δcycle time = 8 Δrpm = 1,330
In raw Euclidean distance, the rpm difference dominates the cycle-time difference

Choosing a Numeric Transformation

Min-max / standardizecomparable scalesfit on training values01Robust scalingmedian and IQRextreme values ignoredcenter = medianspread = IQRLog transformcompress a long tailpositive values onlyUnit-normdirection, not sizesize may carry meaningbefore
Choose a transformation for the comparison the model needs

Encoding, Embeddings and Binning

Category encodingPreserve: membership and any intended ordercolor = redredgreenblue100EmbeddingPreserve: what similarity it has learnedpump P-01near = similarBinningPreserve: thresholds; exact values are lost38.6 s< 3030–40≥ 40
Encoding changes representation; binning also loses within-bin detail

Nominal and Ordinal CategoriesA numeric code can introduce order and spacing into the model

Nominal: no known order

ColorArbitrary code
red1
green2
blue3

Ordinal: a known order

SizeOrdered code
small1
medium2
large3
Color codes invent an order; size codes express an existing order

One-Hot Encoding

sizesmallmediumlargesmall100medium010large001Each column answers one question:“is it this category?”small < medium < largenot expressed by these columnsOrdered levels need ordinal encoding
One-hot represents membership; an ordinal encoding must preserve the intended order

Constructing Features

Written by a personcombine measured fields with domain knowledgeLearned from the dataextract a compact representationBorrowed from a trained modelreuse an embedding; check its fitfeature tablecycle_time / partslatent_1, latent_2embedding[0..63]
A useful feature represents information available when the decision is made

Extraction or SelectionKeep some original columns, or mix all of them into new ones

SELECTION keep some of the original columns 5 columns in, 3 of the same columns out EXTRACTION make new columns from all of them 5 columns in, 3 new columns out
Selection keeps original columns; extraction creates new combinations

Sampling vs Curation

Samplingcriterion: a sampling designInspect: inclusion probabilities and group weightsCurationcriterion: quality and coverage rulesInspect: which groups and examples were removedCuration example: low-quality rows dropped, and group 3 left out entirely
Both select rows; their criteria and resulting biases can differ

Sampling DesignThe selection rule determines which parts of the population survive

MethodHow rows are chosenFailure to check
Simple randomDraw rows at randomSmall groups may be missed
SystematicEvery kth recordA periodic list can align with k
StratifiedRandomly sample at least one per groupKeep the intended group proportions
ClusterSelect whole groupsOne factory may not represent the rest
Check the population structure before choosing the sampling rule

Lossless or Lossy

OriginalcompressedlosslessLossless: exactly the originalcompressedlossyLossy: an approximationCheck: does the discardeddetail matter for the task?
A smaller file is useful only if the retained information still serves the task

The Evaluation Must Match DeploymentDefine what will be new at prediction time before splitting the records

Deployment questionUseful splitWhat a random row split can hide
Predict future eventsEarlier train; later testFuture information reaches training
Predict new machinesHold out whole machinesSame-machine patterns occur on both sides
Predict new customersHold out whole customersRepeated identity makes the test easier
A held-out row can still share an entity or future information with training

Fitting Preprocessing Steps

Training dataFitreplacement value · center and spreadwhich columns to keepLearned parametersstored onceTest and future dataApply the same rulesame value, center, columns✗ never fit on test data: leakage
Fit on training data, then apply the learned transformation consistently

03 · SQL

01

Collection and Exploration

Lec 2–3 · records, labels and patterns

02

Data Preparation

Lec 4–5 · changes to values and representation

03

SQL

Lec 6–8 · read, match and summarize rows

04

NoSQL

Lec 9–10 · models, documents and updates

Outline: SQL I

01

Read a Table

find rows and choose columns

02

Refine a Result

remove repeats, sort and keep top rows

03

Handle Missing Values

choose how absent prices affect the result

04

Write Safely

keep constraints and related changes together

Schema and RowsA schema defines the columns; rows supply their values

Schema: the definition

Product
  pname         TEXT, primary key
  price         REAL
  category      TEXT
  manufacturer  TEXT

Rows: the stored values

pnamepricecategory
Gizmo19.99Gadgets
PowerGizmo29.99Gadgets
SingleTouch149.99Photography
MultiTouch203.99Household
One definition, many product rows

Keys and References

cnamestockpricecountry
GizmoWorks25.0USA
Canon65.0Japan
Hitachi15.0Japan
pnamemanufacturer
GizmoGizmoWorks
PowerGizmoGizmoWorks
SingleTouchCanon
MultiTouchHitachi
A primary key identifies a row; a foreign key requires an existing referenced key
Company facts are stored once; product rows refer to them by key

SELECT, FROM and WHEREThe question names one input table, a row condition and output columns

FROM

Read Product

WHERE

Keep Gadgets

SELECT

Show name and price

SELECT pname, price
FROM Product
WHERE category = 'Gadgets';
Read the table, keep matching rows, then choose columns

Refining a Query Result

Rowsproductpricebagel1.50banana0.50bagel1.50banana1.00Unique valuesproductpricebagel1.50banana0.50banana1.00does each occurrence matter?Ordered by priceproductpricebagel1.50banana1.00banana0.50the ranking the question needsFirst 2 rowsproductpricebagel1.50banana1.00stable order before keeping
Rows, distinct values and a ranked list answer different questions

Missing Values in a Table

partdefectsP-10P-2NULLP-33recorded 0: known, equals zeroNULL: absent, unknown, not applicableWHERE defects = 0partdefectsP-10P-2 ?Decide explicitly whethermissing records belong
Missing is a state of knowledge, not an ordinary numeric value

Safe Changes to Related Facts

Keys and referencesreject duplicate IDs and broken linksidcompanyC1CanonC1Hitachiduplicate key rejectedA deliberate target setcheck which rows will changeitemstatusjournalApaperDpostcardApreview: WHERE status = AA transactionall related changes, or nonestock −1 for item 7order 88 status = paidcommit both, or roll back both
Preserve constraints and change related facts as one unit

Outline: SQL II

01

Match Rows

return one row per matching pair

02

Keep Unmatched Rows

keep the left row when no match exists

03

Test for Matches

keep each outer row if a match exists

04

Stack Results

append compatible result rows

Matching a Product to Its CompanyThe shared company name lets one result contain product and country

Product

pnamemanufacturer
GizmoGizmoWorks

Match the keys

manufacturer = cname

Company

cnamecountry
GizmoWorksUSA
Gizmo and USA meet through the same company key

JOIN on Related Keys

SELECT P.pname,P.price,C.country
FROM Product P JOIN Company C
  ON P.manufacturer=C.cname;
    ↑ match a product to its maker
pnamepricecountry
Gizmo19.99USA
PowerGizmo29.99USA
SingleTouch149.99Japan
MultiTouch203.99Japan
Each product matches its company; the result has four product rows

Pairs, Required Records and Match Existence

Matching pairsone row per pair; an entity can repeatGizmoPowergizmoSingleTouchMultiTouchGizmoWorksCanonHitachiRequired recordskeep every starting row, even without a matchGizmoPowergizmoSingleTouchMultiTouchGizmoWorksCanonHitachiNULLMatch exists?one row per starting entityGizmo✓Powergizmo✓SingleTouch✓MultiTouch✗
Choose the output entity before deciding how records are combined

Matching and Stacking Results

Match records: side by sideStack records: one under anotherproductcidGizmoC1SingleTouchC2+cidcountryC1USAC2JapanproductcidcountryGizmoC1USASingleTouchC2Japankeys must refer to the same kind of entitycityBostonAustin+cityAustinDallasrepeated row:keep or drop?columns must have compatible meanings and types
Matching connects related facts; stacking combines compatible lists

Outline: SQL III

01

Calculate Measures

calculate a quantity from detail rows

02

Summarize Groups

choose the key and keep one row per group

03

Reuse Summaries

use a computed result as another input

04

Keep Detail Rows

add group context without merging rows

Aggregate Functions

FunctionQuestion
COUNT(*)How many purchase rows?
SUM(quantity)How many units?
SUM(price*quantity)How much revenue?
AVG(price)What is the average recorded unit price?
MIN(price), MAX(price)What are the smallest and largest prices?
Choose the expression that measures the quantity named in the question

Quantities and Weights

0102030A: 1 row, total 10B: 9 rows, total 180(10 + 20) / 2 = 15190 / 10 = 19each group counts equallyeach row counts equally
The denominator defines which units receive equal weight

Analytical Result Shapes

Detail input

productprice
bagel1.0
bagel1.5
banana0.5
banana1.0

GROUP BY

productmean
bagel1.25
banana0.75
Grouping turns detail records into one summary row per group

Grouping and Average

One row per product group

SELECT product, AVG(price) AS mean
FROM Purchase
GROUP BY product;

Result

productmean
bagel1.25
banana0.75
The grouping key defines each result row; AVG defines its measure

A Query Produces a Table

Purchase rowsproductpricebagel1.0bagel1.5banana0.5banana1.0Query 1: mean by productproductmeanbagel1.25banana0.75Query 2: keep mean ≥ 1.0productmeanbagel1.25The output of one query is a table, so it can be the input of the next
A query returns a table that another query can use as input

Detail, Summary and Group Context

Detailproductpricebagel1.0bagel1.5banana0.5banana1.0one purchase · 4 rowsGrouped summaryproductmeanbagel1.25banana0.75one product group · 2 rowsDetail with group contextproductpricemeanbagel1.01.25bagel1.51.25banana0.50.75banana1.00.75purchase + its product average · 4 rows
Group information can summarize rows or accompany the original detail rows

04 · NoSQL

01

Collection and Exploration

Lec 2–3 · records, labels and patterns

02

Data Preparation

Lec 4–5 · changes to values and representation

03

SQL

Lec 6–8 · read, match and summarize rows

04

NoSQL

Lec 9–10 · models, documents and updates

Outline: NoSQL I

01

Four Families of NoSQL

choose a model by its access pattern

02

JSON and Semi-Structured Data

read named fields and ordered arrays

03

Between Trees and Tables

convert nesting and repeated values

04

Why NoSQL

check flexibility, scale and guarantees

Four NoSQL FamiliesThe four NoSQL families

SQLRelationalNoSQLK1V1K2V2K3V3Key-valueWide-columnGraphDocument
NoSQL is not one thing: four alternative storage models

Key-Value Records

Request

get("user:1")

Stored pair

key: "user:1"
value: {
  name: "John",
  age: 28
}

Answer

{ name: "John", age: 28 }
A key retrieves its associated value

Sparse Cells Under a Row KeyIn a Bigtable-style model, rows can hold different column qualifiers

user:1

Family: activityValue
page:home09:01
page:product09:03
purchase:1Laptop

user:2

Family: activityValue
page:home09:02
purchase:2TV
Same column family, different populated cells

Property Graphs

FRIENDS_WITHFRIENDS_WITHFRIENDS_WITHFRIENDS_WITHAliceBobCarolDavecity: Busanproperty of a nodesince: 2021property of an edgeNode: an entityEdge: a relationshipProperties: attributesFriends of friends:follow edges, Alice → Bob → Dave
Graph models make relationships and paths explicit

Evolving Document Shapes

One collection

{"name":"John", "age":28}
{"name":"Sue", "phone":"555-0134"}

Different optional fields

John has an age field.

Sue has a phone field.

Both documents can belong to the same collection.

Optional fields may differ; shared fields still need consistent meanings

Semi-Structured Data

Relational: schema declared onceDocument: structure travels with valuesidnamecity1JohnBoston2SueAustinordercust_id1011related facts linked by keys{ "id": 1, "name": "John", "address": { "city": "Boston" }, "orders": [ { "order": 101, "item": "Laptop" }, { "order": 104, "item": "Phone" } ] }nested objects and arrays; field names inside
Semi-structured data carries structure with the values

Trees and Tables

Nested documentRelated tablesorder 101item: Laptopprice: 1200 (copy)item: Phoneprice: 800 (copy)read together in one fetchcopies need coordinated updatesorderpid101P1101P2pidpriceP11200P2800each fact stored oncequeries reconnect them by key
Choose nesting for the access pattern and account for repeated facts

Caveat: Stale CopiesTwo documents cache the current catalog price, but only one copy receives the update

John’s Document
{"name": "John",
 "Orders": [
   {"date": 2004,
    "product": {"prodName": "Gadget",
                "currentCatalogPrice": 12.99}}
 ]}
Sue’s Document
{"name": "Sue",
 "Orders": [
   {"date": 2002,
    "product": {"prodName": "Gadget",
                "currentCatalogPrice": 9.99}}
 ]}
Remedy: reference the current price or maintain every cached copy
One product, two prices: copies need an owner who updates them all

Partitioning and ReplicationSplitting records and copying records solve different problems

Partition: different records

Server A: users 1–100
Server B: users 101–200

Replicate: copies of records

Leader: user 1, value 2
Follower: user 1, value 1
          update pending
Partitions divide the data; replicas copy it and must coordinate updates

Scaling and Operation Guarantees

server 1users A–Hreplicacopy of A–Hserver 2users I–Preplicacopy of I–Pserver 3users Q–Zreplicacopy of Q–Zpartitions split rows; replicas copy thema replica can lag behindCapacitybigger machine, partitions, or bothReading updatescan a read hit a lagging replica?Atomic changesone record or several related onesFailureswhich reads and writes stay available
Choose guarantees for the operation, not from the database label alone

Outline: NoSQL II

01

Documents and Queries

read fields, nesting and paths

02

Finding Documents

choose documents and output fields

03

Summarizing Documents

transform documents through ordered stages

04

Updating Documents

select write targets and verify changes

Collection and Document

Relational terms

TableRowColumn
Productone productprice

Document terms

CollectionDocumentField
inventoryone itemqty
A collection holds documents; each document has named fields

Fields, Objects and Arrays

One inventory document

{
  item: "paper",
  qty: 100,
  size: {h: 8.5, w: 11, uom: "in"},
  tags: ["red", "blank", "plain"],
  instock: [
    {loc: "A", qty: 60},
    {loc: "B", qty: 15}
  ]
}

Levels inside it

FieldValue
qtynumber
sizenested object
tagsarray of strings
instockarray of objects
A field can hold one value, a nested object or an array

Finding Documents

One inventory record

{item:"paper", qty:100, status:"D"}

Two choices

ChoicePurpose
Filter: status DKeep matching documents
Fields: item and qtyChoose what the result shows
Choose documents and output fields before summarizing them

Documents and Nested Entries

One item document

item: "paper"
instock: [
  {loc:"A", qty:60},
  {loc:"B", qty:15}
]

Choose the relevant level

QuestionUnit to inspect
Does one location satisfy both conditions?The same stock entry
How much stock is recorded by location?Each location entry, not each item document
Keep related conditions and measures at the same record level

A Pipeline Changes Its Input

Input documents

{state:"CA", pop:10}
{state:"CA", pop:20}
{state:"NY", pop:40}

Stages and outputs

StageOutput
Keep pop ≥ 20CA:20 · NY:40
Group by state; sum popCA total:20 · NY total:40
Each stage passes its output documents to the next stage

Filter, Group and Measure

zip recordsCA · LOS ANGELESpop 57,110CA · LOS ANGELESpop 51,223CA · SAN FRANCISCOpop 74,633CA · SAN DIEGOpop 34,560NY · BROOKLYNpop 101,572Filterkeep state CAGroupone row per cityMeasureSUM(pop)LOS ANGELES 108,33357,110 + 51,223, not 2 records
Choose the filter, grouping key and measure independently

Safe Document Updates

Preview the selectionstatus A: three itemsitemstatusjournalAnotebookApaperDplannerDpostcardAChange the intended fieldset status to ReviewitemstatusjournalReviewnotebookReviewpaperDplannerDpostcardReviewVerifythree changed; paper and planner stay DitemstatusjournalReviewnotebookReviewpaperDplannerDpostcardReviewAtomic for one document; a multi-document change is not automatically atomic
Preview the targets, apply the change, and verify its scope

Takeaway: Keep the Meaning Through Each Step

  • Define what a record measures and which outcomes were observed
  • Check what preparation changes, estimates or removes
  • Choose a representation and a result that answer the question
← → / Space · S notes · O overview · F fullscreen · number + Enter
Speaker notes