OCR & Text Extraction from Images — Study Pack Summary & Study Notes

📘 Overview

Optical Character Recognition (OCR) is the process of converting text in images into machine-encoded text. OCR systems combine image processing, pattern recognition, and natural language processing (NLP) to detect, segment, recognize, and correct textual content from scanned documents, photos, or screenshots.

🔧 Image Preprocessing

Preprocessing prepares images so recognition models perform reliably. Common steps include grayscale conversion, binarization, deskewing, denoising, contrast enhancement, and resizing. Proper preprocessing reduces noise and normalizes text appearance for the OCR engine.

🧰 Common Preprocessing Techniques

• Binarization: Converts gray images to black-and-white. Techniques include Otsu's method and adaptive thresholding.
• Deskewing: Corrects rotation so text lines are horizontal. Methods use Hough transforms or projection profiles.
• Denoising: Removes salt-and-pepper and Gaussian noise with median or bilateral filters.
• Morphological operations: Use dilation/erosion to close gaps or remove small artifacts.

🏗️ Layout Analysis and Segmentation

Layout analysis identifies regions such as paragraphs, columns, tables, and images. Segmentation breaks text regions into lines, then words, then characters (for character-based OCR). Modern pipelines often use connected components or neural network–based region proposals.

🧠 Recognition Methods

• Classical OCR: Template matching and feature-based classifiers. Works well for printed, high-quality text.
• Statistical/ML OCR: HOG/SIFT features with SVMs or HMMs for sequence modeling.
• Deep learning OCR: Convolutional Neural Networks (CNNs) + Recurrent Neural Networks (RNNs) or Transformers, often using Connectionist Temporal Classification (CTC) or attention-based sequence decoders.

Examples of modern architectures: CRNN (CNN + RNN), Transformer-based OCR, and end-to-end scene text detectors like EAST or CRAFT combined with recognition heads.

🔁 Post-processing and Language Modeling

After raw recognition, post-processing refines output with spell-checkers, language models, and dictionary lookup. Use beam search with language priors or n-gram models to resolve ambiguous outputs. For noisy outputs, apply Levenshtein distance (edit distance) to match probable dictionary entries.

✅ Evaluation Metrics

• Accuracy: ratio of correct characters or words. Useful but often insufficient.
• Precision / Recall / F1 can describe detection of text regions.
• Character-level metrics: Character Error Rate (CER) and word-level Word Error Rate (WER).

Use the following formulas for classification metrics:

$Precision = \frac{TP}{TP + FP}$

$Recall = \frac{TP}{TP + FN}$

$F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}$

CER is typically computed as $\frac{S + D + I}{N}$ where $S$=substitutions, $D$=deletions, $I$=insertions, and $N$=number of reference characters.

⚠️ Common Challenges

• Low-resolution or blurred images reduce recognizability.
• Curved or rotated text requires geometric normalization.
• Complex backgrounds and variable lighting hamper binarization.
• Handwriting recognition remains harder than printed text due to style variability.
• Multilingual text requires language-aware models and fonts.

🧪 Training and Data Augmentation

Augment training data with rotation, scaling, brightness/contrast shifts, blur, and synthetic occlusions. For scene text, add perspective warps and background textures. Balanced datasets across fonts, languages, and imaging conditions improve generalization.

🧩 End-to-end Pipelines

A typical OCR pipeline:

1. Input image capture or scan.
2. Preprocessing: grayscale → binarization → denoise → deskew.
3. Layout analysis / text detection (region proposals).
4. Text line segmentation and cropping.
5. Recognition model (classical ML or deep model).
6. Post-processing with lexicon/language model and confidence thresholds.
7. Output formatting and storage (e.g., searchable PDF).

🛠️ Tools and Libraries

Popular OCR tools include Tesseract, EasyOCR, Google Cloud Vision, AWS Textract, and OpenCV for preprocessing. For deep learning, use PyTorch or TensorFlow with models like CRNN and Transformers.

🧭 Best Practices

• Start with robust preprocessing tuned to your data class.
• Use confidence scores to gate uncertain outputs and human-in-the-loop correction when necessary.
• Combine image-based recognition with language models to correct errors.
• Benchmark with CER/WER and test on in-domain images.

🔚 Summary

OCR blends image processing, recognition models, and NLP-based correction. Success depends on clean input, appropriate model selection, and careful post-processing. For production systems, combine automated pipelines with manual verification for low-confidence cases.