Стоит ли везде применять Kafka?

Стоит ли везде применять Kafka? Когда использовать, когда нет

Краткий ответ: Нет, не везде. Kafka — это мощный инструмент, но не панацея.

1. Когда Kafka НУЖНА

Scenario 1: High-volume event streaming

// Пример: e-commerce платформа с миллионами событий
// Events: page views, clicks, purchases, browsing

@Component
public class EventProducer {
    private final KafkaTemplate<String, Event> kafkaTemplate;
    
    public void publishOrderCreated(Order order) {
        // Миллионы заказов в день — Kafka справляется
        Event event = new OrderCreatedEvent(order);
        kafkaTemplate.send("orders-topic", event);
    }
}

// Advantages:
// - Handles millions of events per second
// - Durable storage (broker side)
// - Replay capability
// - Multiple consumers

Scenario 2: Decoupling сервисов

// Без Kafka (Tight coupling)
OrderService
    ├─> call EmailService
    ├─> call NotificationService
    ├─> call AnalyticsService
    ├─> call WarehouseService
    └─> call RecommendationService

// Если EmailService down → весь OrderService fails

// С Kafka (Loose coupling)
OrderService → Kafka topic "order-created"
                    ↓
          Multiple consumers:
          - EmailService (async)
          - NotificationService (async)
          - AnalyticsService (async)
          - WarehouseService (async)
          
// Если EmailService down → OrderService work!
// Consumer can retry later

@Component
public class OrderService {
    private final KafkaTemplate<String, OrderCreated> producer;
    
    @Transactional
    public Order createOrder(OrderRequest request) {
        Order order = orderRepository.save(new Order(request));
        
        // Send event asynchronously
        producer.send("order-created", new OrderCreated(order));
        // Return immediately, don't wait for consumers
        
        return order;
    }
}

@Component
public class EmailConsumer {
    @KafkaListener(topics = "order-created")
    public void sendConfirmationEmail(OrderCreated event) {
        emailService.send(event.getEmail(), "Order confirmed");
    }
}

Scenario 3: Event sourcing и audit trail

// Kafka как source of truth для events

@Component
public class OrderEventStore {
    private final KafkaTemplate<String, DomainEvent> producer;
    
    public void saveEvent(Order order, DomainEvent event) {
        // All events are immutable and stored
        producer.send("order-events", event);
    }
}

// Later: можем воспроизвести полную историю
public void rebuildOrderState() {
    // Read all events from Kafka from beginning
    // Replay them to reconstruct current state
    List<DomainEvent> events = readAllEventsFromKafka();
    Order order = new Order();
    events.forEach(event -> order.apply(event));
}

2. Когда Kafka НЕ НУЖНА

Scenario 1: Simple synchronous operations

// Плохо: использовать Kafka
@RestController
public class BadController {
    @PostMapping("/users")
    public ResponseEntity<User> createUser(@RequestBody UserRequest request) {
        kafkaTemplate.send("user-created", request);
        return ResponseEntity.ok(new User());  // Don't wait for consumer
        // Client doesn't know if user was created!
    }
}

// Хорошо: просто вызвать сервис
@RestController
public class GoodController {
    private final UserService userService;
    
    @PostMapping("/users")
    public ResponseEntity<User> createUser(@RequestBody UserRequest request) {
        User user = userService.createUser(request);  // Synchronous
        return ResponseEntity.ok(user);
    }
}

Scenario 2: Low volume, one-time operations

// Не нужна Kafka для такого:

@Component
public class OnboardingService {
    private final EmailService emailService;
    
    public void onboardNewCompany(CompanyRequest request) {
        // Просто вызвать напрямую
        emailService.sendWelcomeEmail(request.getContactEmail());
        // Kafka overhead не нужен
    }
}

// Если volume малый — используй RabbitMQ или простой REST call

Scenario 3: Real-time request-response

// Bad: Kafka
// User orders pizza → Event to Kafka → Consumer prepares → 
// Send SMS → User waits minutes for response

// Good: REST API
POST /orders → OrderService → return 200 OK immediately
User gets response in milliseconds

3. Альтернативы Kafka

4. Типичные ошибки использования Kafka

Ошибка 1: Over-engineering

// ПЛОХО: использовать Kafka для всего

@PostMapping("/search")
public void search(@RequestParam String query) {
    kafkaTemplate.send("search-events", query);
    // Then in consumer:
    searchService.search(query);
    // Result sent to client via WebSocket???
}

// ХОРОШО: просто REST
@PostMapping("/search")
public SearchResult search(@RequestParam String query) {
    return searchService.search(query);
}

Ошибка 2: Забыть про latency

// Kafka добавляет latency
// Broker → network → consumer → processing → response
// Min latency: 10-100ms (даже в локальной сети)

// Если нужна latency < 1ms:
DO NOT use Kafka ❌
use in-process queue or cache ✅

Ошибка 3: Неправильное разбиение на topic'ы

// ПЛОХО: слишком много topic'ов
topics:
  - order-created-v1
  - order-created-v2
  - order-updated
  - order-deleted
  - order-confirmed
  - order-shipped
  // ... 50 topic'ов

// ХОРОШО: логическое разбиение
topics:
  - order-events (contains все order events)
  - payment-events
  - inventory-events

5. Простое правило: вопросник

Нужна ли Kafka?

1. High volume (>1000 events/sec)?
   YES → Possible candidate
   NO → Go to question 2

2. Multiple async consumers?
   YES → Good use case
   NO → Go to question 3

3. Need event replay/history?
   YES → Good use case
   NO → Go to question 4

4. Need to decouple services?
   YES → Kafka or RabbitMQ
   NO → Probably don't need it

5. Latency requirements < 10ms?
   YES → DON'T use Kafka ❌
   NO → Kafka is OK

6. Operations team can handle it?
   NO → Use managed service or simpler tool
   YES → Kafka OK

6. Real-world example: когда Kafka нужна

// Платформа для торговли акциями
// - Миллионы трейдов в день
// - Нужна история всех трейдов
// - Нужны async обработчики:
//   * Risk monitoring
//   * Compliance checking
//   * Market data update
//   * User notifications
//   * Analytics

@Component
public class TradeService {
    private final KafkaTemplate<String, TradeEvent> producer;
    
    public void executeTrade(TradeRequest request) {
        Trade trade = tradeRepository.save(new Trade(request));
        producer.send("trades", new TradeExecuted(trade));
    }
}

@Component
public class RiskMonitoring {
    @KafkaListener(topics = "trades")
    public void monitorRisk(TradeExecuted event) {
        if (riskCalculator.exceeds(event)) {
            alertManager.sendAlert("Risk limit exceeded");
        }
    }
}

// In this case: Kafka is PERFECT ✅

7. Real-world example: когда Kafka не нужна

// Simple blog platform
// - Create post
// - Update post
// - Delete post
// - Only one backend service

@RestController
public class PostController {
    private final PostService postService;
    
    @PostMapping("/posts")
    public PostDTO createPost(@RequestBody PostRequest request) {
        Post post = postService.createPost(request);
        return PostDTO.from(post);
    }
}

// Here: Kafka is OVERKILL ❌
// Use simple REST or database transaction

Резюме

Kafka НУЖНА когда: ✅ High volume events (>1000/sec) ✅ Multiple async consumers ✅ Event sourcing / replay capability ✅ Service decoupling needed ✅ You can afford operational complexity

Kafka НЕ НУЖНА когда: ❌ Low volume events ❌ Real-time request-response needed ❌ Single consumer ❌ Latency < 10ms required ❌ Team can't operate it

Практический совет: Start simple, add Kafka only when you have the problem it solves. Most systems don't need Kafka.