IMEN891K / IDSC501 / AXGS-501 · Data Engineering · Fall 2026

2. Data Collection

Sangwoo Mo · POSTECH
Industrial & Management Engineering

Where We AreCollection and preparation shape the data that later stages can use

PART I · Collection and Preparation PART II PART III Collect Lec 2 Explore Lec 3 Preprocess Lec 4–5 Storage Lec 6–13 Monitoring and Updates Lec 14–17 every later stage can only use what you collect here
You have a problem worth solving
You have no data

Now what?

Data for Your TaskA public dataset can help, but it cannot supply your missing observations

ImageNet image collection
Machine inspection

Which parts need inspection?

Inputs: vibration, current, tool

Label: the inspection result

Collect both for the same part

A useful dataset must connect the input to the outcome

The record is the competenceGoogle: queries and clicks provide feedback that an algorithm alone cannot supply

Google search
Search use generates new queries and feedback

The record is the moatQANDA: product use generates observations that a downloaded dataset does not contain

QANDA
Each student photo records a moment of difficulty and creates another observation

The record is the strategySensor choices determine which aspects of the world become data

Sensor types and coverage zones around a vehicle
Each sensor captures different information

Case study: Waymo vs TeslaDifferent sensor designs produce different training evidence

Collection choiceWaymoTesla Vision
SensorsCameras + lidar + radarCamera-based perception
Distance evidenceLidar range measurementsDepth estimated from images
Data workAlign readings across sensorsTrain and validate visual depth
A measurement and a model estimate need different checks

Outline: Data Collection

01Define the RecordWhat does one row hold?one row = one part02Build the Collection PathWhere does each value come from?reuse, observe, generate03Collect Reliable LabelsHow do we get reliable labels?annotate, observe, validatewhat comes back changes what you collect next

Define the Record

01Define the RecordWhat does one row hold?one row = one part02Build the Collection PathWhere does each value come from?reuse, observe, generate03Collect Reliable LabelsHow do we get reliable labels?annotate, observe, validatewhat comes back changes what you collect next

Record DesignDefine the meaning, choose the representation, and set the permitted use

MeaningMeasurement

Definition and measurement procedure

→
RepresentationRecord structure

Unit, resolution and metadata

→
UseBoundaries

Purpose and identifiability

Label DefinitionsDifferent definitions create different labels for the same parts

ReaderDefect definitionSame 400 parts
QC inspectorOutside drawing tolerance40
Line supervisorCustomer returned the part12
SalesComplaint call received3
A model learns the label definition used to build its dataset

Constructs and MeasuresOperationalization turns a concept into a measurement procedure

ConstructDefect

The property we want to identify

→
OperationalizationInspection rule v3

Measure against drawing tolerances

→
MeasurePass / defect

The recorded inspection result

Validity
Does the procedure measure the intended property?

Reliability
Do repeated measurements agree?

Agreement alone does not show that the right property was measured

Unit of AnalysisChoose what one row represents before choosing its columns

One row representsUseful forCan identify a defective part?
One partPart inspectionYes, with part_id
One machine-minuteMachine monitoringOnly with an additional part-level link
One work orderYield reportingNo individual part result
The task determines the grain of the record

Measurement ResolutionA summary can discard information that a later task needs

20 kHz raw waveform spectrum bearing fault visible RMSover 1 min 1-minute RMS (root mean square) spectrum fault frequency lost

Context Around the ValueKeep the unit, instrument and event time with the value

4.7unit and ratemm/s RMS, derived at 1 Hzinstrumentsensor A-17, firmware 2.3event time2026-10-04 09:14:01.120 KSTlabel definitiondefect rule v3permitted purpose and licensepurpose: inspection only4.7 of what? when? measured how?
Metadata makes the measurement interpretable and reusable

Dataset DocumentationA dataset needs enough context for someone else to interpret and use it

Dataset: part inspectionsDocument with the data
MeaningOne row per part; defect defined by rule v3
CollectionWhich machines, dates and inspection policy
Use and limitsInspection purpose; uninspected parts have no label
MaintenanceNamed owner; update and guideline versions
A datasheet records choices that column names cannot explain

Demo: Reading a RecordA measurement needs an entity, a time, a unit and a rule

FieldRecorded value
part_idP-104
measured_at09:14:02
value38.6
unitmissing
sourceCNC-02
labeldefect
1. Which fields are missing?
2. Can two inspectors reproduce this label?

Answer: A Reproducible RecordStore the measurement meaning and the labeling rule

FieldRecorded value
part_id / measured_atP-104 / 09:14:02
measurement / value / unitcycle_time / 38.6 / s
sourceCNC-02 / cycle log
label / rule_versiondefect / inspection-v3
The value becomes interpretable when its context is recorded

Privacy: Purpose LimitationSpecify the purpose and legal basis before collecting personal data

Line cameraconsent: safetySafety monitoringthe declared purpose✓Scoring workersa new purpose✗needs a valid basis for the new use
Failure: safety cameras on the line, later used to score workers. Consent covered safety, not scoring

Caveat: AnonymizationRemoving names does not prevent identification through other fields

Released dataset · name removed ZIPBirth dateSexDiagnosis 376731965-07-31FHypertension 376731971-02-14MAsthma 376751965-03-02FDiabetes Public list · has names NameZIPBirth dateSex J. Kim376731965-07-31F join J. Kim has hypertension k-anonymity Every (ZIP, birth date, sex) combination must appear in at least k rows Generalize to reach it: ZIP 376**, birth year 1960s. Each step costs precision
Failure: drop the name column and call it anonymous. A public list with matching fields can join it back

Build the Collection Path

01Define the RecordWhat does one row hold?one row = one part02Build the Collection PathWhere does each value come from?reuse, observe, generate03Collect Reliable LabelsHow do we get reliable labels?annotate, observe, validatewhat comes back changes what you collect next

Data AcquisitionCollect observations, or create training examples from existing evidence

How will you obtain the data?Collect observationsreuse records or create new observationsCreate training examplesadd annotations or generate inputsImporta record already existspart_no, pricefrom the supplier APIInstrumentno record until you make onevibration, current: sensortool change: operator entryAnnotateexisting input, new candidate labelcandidate defect labelfrom an image modelSynthesizenew inputs are generatedrare chatter patternsfrom a simulatorKeep the source; keep the original when adding labels or generating variants

Observe Real Events

02 · Build the Collection Path
How will you obtain the data?axis: new observations or examples from existing evidenceObserve Real Eventsis the event already recorded somewhere?Importa record exists:API, CSV, crawlInstrumentno record yet:sensor, person, programGenerate Training Exampleschange the annotation or the input?Annotateevidence fixed,annotation changesSynthesizenew inputs:simulator, diffusion

Existing RecordsStart with the system that already records the event

SourceRecord to reuseAccess route
Supplier systemPart specification, priceFile export or API
Production systemPart, tool, program, operatorDatabase view or API
Machine controllerSignals and alarm eventsLog file or event stream
The source tells you what was recorded; the interface tells you how to retrieve it

API or CrawlerPrefer an explicit interface; page extraction also depends on the page layout

API their system a contractversioned, documented your rows An explicit interface can be versioned and checked CRAWLER their system a page for humansredesigned on their schedule your selector 0 rows no error raised A redesign breaks extraction without an HTTP error

Demo: Crawling a Supplier PortalExtract prices, change the page layout, then repair the selector

1. Does a successful request guarantee extracted rows?   2. Repair the selector after redesign
# extraction from one fetched page
r    = requests.get("https://supplier-portal.example.com/products?page=1")
rows = BeautifulSoup(r.text).select(".product .price")

      

    

    

  







Answer: Detect a Broken ExtractionA successful request does not guarantee usable records




RunResponseExtracted pricesAction
Before redesign200 OK18Validate fields
After redesign200 OK0Investigate the drop
Selector repaired200 OK18Compare with the last good run
.product .price → inspect HTML → .product .pdp-price__amount
Check counts, required fields and types before accepting the run






  
Copyright: Three GatesAccess, site terms and rights to reuse are separate checks

  

  
Public pageyou can download itrobots.txta technical requestTerms of usea contractCopyrightthe lawtrainPassing one gate does not pass the othersTraining rights depend on the data, use and jurisdiction

  
Failure: a crawled dataset with no documented basis for use. Record the license or the exception you rely on







Creating New ObservationsAn event becomes data only when something records it




SoftwareAlarm acknowledged
Instrument the button press
SensorVibration measured
Mount and calibrate the sensor
PersonDefect judged
Apply the inspection guideline
Logging needs instrumentation; physical measurements and judgments need it too






  
Sensor PlacementSensor coverage depends on placement as well as the number of sensors

  

  
Coverage with three, four and five sensors

  
Going from 3 to 5 sensors changes what is detectable, not just the accuracy






  
Generate Training Examples

  

  
02 · Build the Collection Path
How will you obtain the data?axis: new observations or examples from existing evidenceObserve Real Eventsis the event already recorded somewhere?Importa record exists:API, CSV, crawlInstrumentno record yet:sensor, person, programGenerate Training Exampleschange the annotation or the input?Annotateevidence fixed,annotation changesSynthesizenew inputs:simulator, diffusion







Weak SupervisionLabeling functions provide noisy votes that can agree, conflict or abstain




PartVibration ruleReturn reportImage modelVotes
4471defect—defectagree
4472——passone source
4473defect—passconflict
4474———uncovered
Labeling functions → label model → candidate training labels
Coverage: 3 of 4 parts receive a vote. Check conflicts on reviewed examples






  
Synthesizing the Rare ClassAugmentation changes an input while preserving its label; generation proposes new examples

  

  
Diffusion-based data augmentation

  
Variations supplement scarce real examples; their realism and value still need testing






  
Confidentiality: Trade SecretsCheck whether the data may leave the organization before choosing a cloud service

  

  
The plantProcess datayield, defects, recipesCloud APIoutside the plant✗ data may not leaveThe modelshipped to the data✓ the model goes in

  
Failure: pipeline built on a cloud service, then the confidentiality agreement forbids upload. Rebuild inside, or drop that route






  
Security: Attack SurfaceLimit access at every point where data is collected, transferred or stored

  

  
Sensor!default passwordGateway!unpatched firmwareCrawler!key in a notebookLanding bucket!left publicWarehouseEach one needs encryption, least privilege and a named owner

  
Failure: a bucket left public for a demo, indexed by a search engine before anyone notices






  
Collect Reliable Labels

  

  
01Define the RecordWhat does one row hold?one row = one part02Build the Collection PathWhere does each value come from?reuse, observe, generate03Collect Reliable LabelsHow do we get reliable labels?annotate, observe, validatewhat comes back changes what you collect next







Sources of LabelsA label can come from deliberate annotation or from an observed outcome




SourceExampleMain collection problem
Human annotationAn inspector judges a partMake the guideline reproducible
Natural labelA delivered shipment has an arrival timeRecord and link the outcome
Model-generated labels remain candidates until checked
Both sources need a definition and a quality check






  
Human Annotation

  

  
03 · Collect Reliable Labels
Where does the label come from?axis: deliberate annotation or observed outcomeHuman Annotationa person applies a guidelineAgreementtwo readers,same guidelineLabel Budgetwhere the modelis unsure, plus auditNatural Labelsan outcome is observed during useObservationlate and selectedoutcomesMissing Statesnot inspectedis not a negative






  
Running a Labeling ProjectPilot the guideline, resolve disagreements, then expand the labeling effort

  

  

  
1 Write the guidelinerule v3 + borderline photos2 Pilot2 labelers, same 100 parts3 Measure agreementκ, next slide4 Adjudicatedecide the disagreements5 Label at scalekeep a random auditdecisions go back into the guideline as examples, then pilot again

  

    
In-house experts
Use domain expertise for ambiguous cases and audits

    
Vendor or crowd
Use when the task can be taught with clear examples

    
Tooling
Label Studio / CVAT: configure labeler, time and guideline tracking

  

  





Demo: Inspector AgreementTwo inspectors each mark five defects among the same 100 parts
B: defectB: pass
A: defect14
A: pass491
1. What is the overall agreement?
2. How many defects did both inspectors identify?




  
Answer: Inspector AgreementCohen’s kappa corrects agreement for chance matches

  

  

    

      

        
B: defectB: pass

        
A: defect14

        
A: pass491

      

      
Two inspectors, 100 parts. Each calls 5 defects

    

    

      
κ = (po − pe) / (1 − pe)

      
po = (1 + 91) / 100 = 0.92 observed

      
pe = 0.05·0.05 + 0.95·0.95 = 0.905 by chance

      
κ = 0.015 / 0.095 = 0.16

      
    

  

  
One shared defect. Adjudicate, revise the rule, measure again





Active Learning: Uncertainty SamplingTarget uncertain cases for learning; audit independently to find blind spots
Partp(defect)What a new label tells us
44710.99Model is confident; it can still be wrong
44720.50Model is uncertain; request a label
44730.80Less uncertain under this model
44740.02A random audit can check confident predictions
Model-guided labeling and random auditing answer different questions





  
Natural Labels

  

  
03 · Collect Reliable Labels
Where does the label come from?axis: deliberate annotation or observed outcomeHuman Annotationa person applies a guidelineAgreementtwo readers,same guidelineLabel Budgetwhere the modelis unsure, plus auditNatural Labelsan outcome is observed during useObservationlate and selectedoutcomesMissing Statesnot inspectedis not a negative






  
Predictions Without OutcomesPrediction logs alone cannot tell you whether the predictions were useful

  

  
Inspection recommendations, 12 monthsdatepart_idscorerecommendationoutcome2025-10-0334710.91inspect—2025-10-0344800.88inspect—2025-11-1251020.93inspect—2026-01-2062100.86inspect—2026-03-0470040.90inspect—2026-06-1788310.95inspect—2026-09-3099160.89inspect—… 2,140 rowsThe operator screenALARM  part 9916  score 0.89recommend: inspectNo button to record what happened nextThe operator inspects, or does notThe part is fine, or it is notNothing writes that backA year later, this log cannot say if the model works





Feedback Loop LengthAn outcome can arrive after the prediction record has already been stored
Decision recorded
part_id: P-104
09:14 · inspect
Outcome pending
Inspection not finished
No label yet
Outcome linked
09:20 · pass
label = 0
Other examplesPossible feedback delay
Recommendation → clickMinutes
Inspection → resultDays
Fraud → disputeWeeks
A pending outcome is not a negative outcome




Selected OutcomesThe inspection policy controls which outcomes become labels
100 produced parts
20 selected for inspection
Labels will be observed
80 shipped without inspection
Outcomes unknown
They are not recorded as pass
Random audit of 5
25 labels in total
75 outcomes remain unobserved
Audit outside the model-selected sample to observe what it missed





  
Missing OutcomesKeep pending, unobserved and negative outcomes distinct

  

  
decidedrow written at 09:14:02pendingsent to inspectionnot inspectedshipped without a lookobserved: passlabel = 0observed: defectlabel = 1still missingno label after 30 daysresult arrivesnever this arrownot inspected ≠ passA model trained on "label or 0" learns that skipped parts are good





Linking by Identity and TimeThe same identifier is necessary; the event sequence still matters
Earlier observation
P-104 · 08:30
defect

Not this decision’s outcome
Decision
P-104 · 09:14
inspect
Later outcome
P-104 · 09:20
pass

Match identity and time
A prior inspection cannot be the outcome of a later decision




Demo: Linking Inspection OutcomesUse identifiers and event times instead of row order
Predictions
part_iddecided_atdecision
P-10409:14inspect
P-10509:15ship
P-10609:16inspect
Inspection log
part_idinspected_atresult
P-10609:22pending
P-10409:20pass
P-10408:30defect
1. Which inspection belongs to P-104’s decision?
2. Does P-105 have a negative label?



Answer: Observed and Missing LabelsPolicy: link the first inspection episode after each decision
part_idlinked inspectionstatelabel
P-10409:20observed pass0
P-105nonenot inspectedNULL
P-10609:22pendingNULL
Missing feedback is a state, not a negative label





Goodhart’s LawA reward tied to a reported metric can change how people report it








from here, the plant pays a bonus for a low defect rate

defects
the plant reports

defects
customers send back
time →


Reported defects fall while returns stay flat; audit whether reporting behavior changed






  
Fairness: Historical SelectionLabels reflect past access and decisions, not only the property we want to predict

  

  
Past hiring choicesLabelsjob performance: hired people onlyModel trained on themSelects future applicantsmay repeat the same exclusionsbecomes the next history

  
Job performance is unobserved for rejected applicants; rejection is not evidence of poor performance






  
Closing the LoopUse verified outcomes and audits to improve the next collection cycle

  

  
Decide and collectthe record of Part 01Use itinspect 20; audit 5 of the other 80Outcomes and auditlabels and missing statesFind the gapa tool condition never coveredcollect that condition nextone turn of the loop = one inspection cycle






  
Takeaway: Data CollectionDefine what to observe, obtain the evidence, and check the labels

  

  
01Define the RecordDefine the unit, measurement and permitted use
02Build the Collection PathReuse records, instrument events, or generate examples
03Collect Reliable LabelsCheck agreement, delayed outcomes and selection bias










  
  
  



← → / Space · S notes · O overview · F fullscreen · number + Enter


Speaker notes