> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/alblandino/tokenizador/llms.txt
> Use this file to discover all available pages before exploring further.

# Introduction

> Visualize how AI models process and tokenize your text in real-time

## Welcome to Tokenizador

Tokenizador is a professional, real-time AI tokenization visualization tool that helps you understand how different AI models process and tokenize text. With support for **48 models** from leading providers like OpenAI, Anthropic, Google, Meta, and more, you can visualize tokenization, estimate costs, and compare models all in one place.

<CardGroup cols={2}>
  <Card title="Quick Start" icon="rocket" href="/quickstart">
    Get started with Tokenizador in minutes
  </Card>

  <Card title="How to Use" icon="book-open" href="/guides/how-to-use">
    Learn how to use the tokenization analyzer
  </Card>

  <Card title="Supported Models" icon="microchip" href="/guides/supported-models">
    View all 48 supported AI models
  </Card>

  <Card title="API Reference" icon="code" href="/api/token-analyzer">
    Explore the JavaScript API
  </Card>
</CardGroup>

## Key Features

<CardGroup cols={2}>
  <Card title="Real-Time Analysis" icon="bolt">
    Tokens are calculated as you type, with instant feedback on tokenization
  </Card>

  <Card title="48 Models Supported" icon="brain">
    Support for OpenAI, Anthropic, Google, Meta, Mistral AI, and many more providers
  </Card>

  <Card title="Cost Estimation" icon="dollar-sign">
    Automatically calculates the estimated cost per model based on current pricing
  </Card>

  <Card title="Interactive Visualization" icon="eye">
    Color-coded token display with detailed token lists and statistics
  </Card>

  <Card title="Context Warnings" icon="triangle-exclamation">
    Get warnings when your text exceeds model context limits
  </Card>

  <Card title="Export Results" icon="download">
    Export analysis results in JSON, CSV, or TXT formats
  </Card>
</CardGroup>

## What is Tokenization?

Tokenization is the process of breaking down text into smaller units called "tokens" that AI models can understand. Different models use different tokenization strategies, which affects:

* **Token count** - How many tokens your text uses
* **Cost** - API costs are typically charged per token
* **Context limits** - Maximum text length a model can process

<Tip>
  Understanding tokenization helps you optimize your prompts and manage API costs more effectively.
</Tip>

## Live Demo

Visit [tokenizador.alblandino.com](https://tokenizador.alblandino.com) to try the tool in your browser. No installation required!

## Who is this for?

<AccordionGroup>
  <Accordion title="AI Developers" icon="code">
    Optimize your prompts, estimate API costs, and understand how different models process your text
  </Accordion>

  <Accordion title="Content Writers" icon="pen">
    See how your content is tokenized and ensure it fits within model context limits
  </Accordion>

  <Accordion title="Data Scientists" icon="chart-line">
    Compare tokenization strategies across models and analyze text processing patterns
  </Accordion>

  <Accordion title="Students & Researchers" icon="graduation-cap">
    Learn about tokenization and explore how different AI models work under the hood
  </Accordion>
</AccordionGroup>

## Architecture

Tokenizador is built with a modern, modular architecture:

* **Vanilla JavaScript** - No framework dependencies, fast and lightweight
* **Tiktoken Library** - Industry-standard tokenization for maximum accuracy
* **Modular Design** - Separation of concerns with services, controllers, and utilities
* **Responsive UI** - Works perfectly on desktop, tablet, and mobile

<Card title="Explore the Architecture" icon="sitemap" href="/architecture/overview">
  Learn about the modular architecture and component design
</Card>

## Get Started

Ready to start visualizing tokenization? Follow our quickstart guide to get up and running in minutes.

<Card title="Quickstart Guide" icon="play" href="/quickstart" horizontal>
  Start using Tokenizador in 3 simple steps
</Card>