← All work

Hebrew sentiment classification

Academic NLP work: a LoRA fine-tuned DictaBERT for Hebrew sentiment, compared against GPT-4o zero/few-shot and RAG baselines, and a GPT-style transformer built from scratch in PyTorch.

Type
Academic, NLP course
Stack
PyTorch, HuggingFace Transformers, LoRA
Data
43,600 examples, Macro F1
Robustness heatmaps for Hebrew sentiment methods under spelling variation

Overview

Hebrew is less well served by general-purpose models than English. The question was whether a small, fine-tuned Hebrew model could do better on sentiment than prompting a large general model.

What I built

  • Fine-tuning. DictaBERT fine-tuned with LoRA on 43,600 labelled examples.
  • Baselines. GPT-4o zero-shot, few-shot and a RAG setup, all evaluated with Macro F1.
  • Result. The fine-tuned model outperformed the GPT-4o and RAG baselines.

Transformer from scratch

Separately, I implemented a GPT-style transformer from scratch in PyTorch, including self-attention, causal masking and LayerNorm, to understand the architecture below the library level.

Screenshots

Tokenizer fertility change for each perturbation family
Tokenizer fertility change for each perturbation family.