Machine Learning Yearning

• Why Machine Learning Strategy
• How to use this book to help your team
• Prerequisites and Notation
• Scale drives machine learning progress
• Your development and test sets
• Your dev and test sets should come from the same distribution
• How large do the dev/test sets need to be?
• Establish a single-number evaluation metric for your team to optimize
• Optimizing and satisficing metrics
• Having a dev set and metric speeds up iterations
• When to change dev/test sets and metrics
• Takeaways: Setting up development and test sets
• Build your first system quickly, then iterate
• Error analysis: Look at dev set examples to evaluate ideas
• Evaluating multiple ideas in parallel during error analysis
• Cleaning up mislabeled dev and test set examples
• If you have a large dev set, split it into two subsets, only one of which you look at
• How big should the Eyeball and Blackbox dev sets be?
• Takeaways: Basic error analysis
• Bias and Variance: The two big sources of error
• Examples of Bias and Variance
• Comparing to the optimal error rate
• Addressing Bias and Variance
• Bias vs. Variance tradeoff
• Techniques for reducing avoidable bias
• Error analysis on the training set
• Techniques for reducing variance
• Diagnosing bias and variance: Learning curves
• Plotting training error
• Interpreting learning curves: High bias
• Interpreting learning curves: Other cases
• Plotting learning curves
• Why we compare to human-level performance
• How to define human-level performance
• Surpassing human-level performance
• When you should train and test on different distributions
• How to decide whether to use all your data
• How to decide whether to include inconsistent data
• Weighting data
• Generalizing from the training set to the dev set
• Identifying Bias, Variance, and Data Mismatch Errors
• Addressing data mismatch
• Artificial data synthesis
• The Optimization Verification test
• General form of Optimization Verification test
• Reinforcement learning example
• The rise of end-to-end learning
• More end-to-end learning examples
• Pros and cons of end-to-end learning
• Choosing pipeline components: Data availability
• Choosing pipeline components: Task simplicity
• Directly learning rich outputs
• Error analysis by parts
• Attributing error to one part
• General case of error attribution
• Error analysis by parts and comparison to human-level performance
• Spotting a flawed ML pipeline
• Building a superhero team - Get your teammates to read this