Kafka Retention Calculator Documentation

A minimalist web application for calculating optimal retention.bytes settings for Kafka topics based on your storage configuration and topic specifications.

Features

How It Works

The calculator determines optimal retention.bytes settings by:

1. Calculating Total Storage: Multiplies PVC count by PVC size to determine total available storage
2. Applying Kafka Overhead: Reserves configurable percentage of total storage for Kafka internal operations, log segments, and system files
3. Weighted Storage Distribution: Allocates remaining storage based on topic weights and partitions
4. Accounting for Replication: Adjusts retention based on replication factor requirements
5. Providing Metrics: Shows storage utilization, retention per partition, and summary statistics

Retention Time Calculation (Beta)

When you provide event size and volume information, the calculator can estimate how long data will be retained based on your storage configuration.

Important Notes:

• Event Volume per Topic: The retention time calculation always refers to the combined event rate per second for each individual topic. It does not account for events that might be selectively routed or split between multiple topics.
• Topic-Specific Calculation: Each topic's retention time is calculated independently based on its allocated storage and the event volume you specify for that topic.
• Formula: Retention Time = (Topic Storage in Bytes) / (Event Rate per Second × Event Size in Bytes × Replication Factor)

Example:

If you have 400,000 events per second with 1KB event size, replication factor 2, and 2,400GB of storage allocated to a topic, the retention time would be approximately 50 minutes.

Usage

1. Storage Configuration

• Number of PVCs: Total number of Persistent Volume Claims in your Kafka cluster (equals number of Kafka pods)
• PVC Size (GB): Size of each PVC in gigabytes

2. Topic Configuration

For each topic, specify:

• Topic Name: Descriptive name for your topic
• Partitions: Number of partitions for the topic
• Replication Factor: Replication factor (1-3, typically 3 for production)
• Weight (%): Storage allocation priority (1-100%). Higher weights get more storage allocation

3. Results

The application provides:

• retention.bytes: The calculated retention setting in bytes (raw value for Kafka configuration)
• Retention (GB): Human-readable retention size in gigabytes
• Weight Allocation: Percentage of available storage allocated to this topic
• Storage per Partition: Available storage per partition
• Storage Utilization: Percentage of available storage used by each topic
• Summary: Overall storage statistics including Kafka overhead and utilization

Example Configuration

Storage Setup:

• 3 PVCs × 100 GB = 300 GB total storage
• Kafka Overhead: 60 GB (20% reserved)
• Available for Topics: 240 GB

Topic Setup:

• Topic: user-events (6 partitions, RF=3, Weight: 80%)
• Topic: system-logs (3 partitions, RF=3, Weight: 20%)

Result:

• user-events: retention.bytes = 22,906,492,682 (≈21.33 GB) - 80% allocation
• system-logs: retention.bytes = 5,726,623,171 (≈5.33 GB) - 20% allocation

Getting Started

1. Open index.html in your web browser
2. Configure your storage settings
3. Add your Kafka topics with appropriate weights
4. View the calculated retention settings
5. Use the retention.bytes values (in bytes) in your Kafka topic configurations

Technical Details

Calculation Formula

Total Storage = PVC Count × PVC Size (GB)
Available Storage = Total Storage × 0.8 (20% overhead reserved for Kafka)
Weighted Storage per Topic = Available Storage × (Topic Weight / Total Weight)
Storage per Partition = Weighted Storage per Topic / Topic Partitions
Retention per Partition = Storage per Partition / Replication Factor
Retention Bytes = Retention per Partition × 1024³

Browser Compatibility

• Chrome 60+
• Firefox 55+
• Safari 12+
• Edge 79+

File Structure

kafka-kalkulator/
├── index.html          # Main application page
├── readme.html         # This documentation page
├── styles.css          # Styling and responsive design
├── script.js           # Calculation logic and interactivity
├── README.md           # GitHub documentation
├── LICENSE             # MIT License
└── package.json        # Project metadata

Contributing

This is a simple, single-page application designed for Kafka administrators. Feel free to fork and modify for your specific needs.

License

MIT License - feel free to use and modify as needed. See the LICENSE file for details.