Which Loss Function Do LLMs use?
Exploring Cross-Entropy Loss in Large Language Models.
Technical
RSS feedPosts in 2025
- $What is Matryoshka Representation Learning (MRL)?What is Matryoshka Representation Learning (MRL)?Nesting Power and Flexibility into ML Embeddings
- $2-Hour Streamlit Workshop2-Hour Streamlit WorkshopBeginners with a basic understanding of Python. No web development experience is required.
- $UE8M0 FP8 Number FormatUE8M0 FP8 Number FormatTraining LLMs without H100 using UE8M0 FP8 number format.
- $Understanding ML Numerical FormatsUnderstanding ML Numerical FormatsUnderstanding INT4, INT8, FP16, BF16, and TF32 formats in machine learning - their precision, speed, and memory trade-offs for training and inference.
- $What do GPT-OSS and Gemma 3 really offer?What do GPT-OSS and Gemma 3 really offer?GPT-OSS and Gemma 3: two new small-but-powerful language models pushing the boundaries.
- $What are Positional Embeddings?What are Positional Embeddings?The mathematical technique that teaches AI models where each word sits in a sequence.
- $Words, Tokens and EmbeddingsWords, Tokens and EmbeddingsHow language models convert token IDs into meaningful vector representations that capture semantic relationships.
- $Subword Tokenization AlgorithmsSubword Tokenization AlgorithmsUnderstanding the algorithms behind tokenization in Large Language Models.
- $What is LLM Inference?What is LLM Inference?Understanding how Large Language Models generate text through the inference process.
