Что использовал из инструментов?

Инструменты, которые я использовал за карьеру

Я работал с широким спектром инструментов. Расскажу про каждый, его сильные стороны и где я его применял.

Big Data Processing

Apache Spark (4+ года)

• Batch: преобразования 100TB+ датасетов
• Streaming: real-time обработка событий
• ML: подготовка признаков для моделей

# Типичный Spark job
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum, count

spark = SparkSession.builder.appName("sales_analysis").getOrCreate()

df = spark.read.parquet("/data/raw/events")
result = df \
    .filter(col("event_date") >= "2026-03-01") \
    .groupBy("product_category", "country") \
    .agg(sum("amount"), count("*")) \
    .write.parquet("/data/processed/daily_sales")

Apache Flink (2 года)

• Real-time streaming (stateful processing)
• Event-driven architectures
• Complex windowing operations

# Flink job для обработки событий в реальном времени
from pyflink.datastream import StreamExecutionEnvironment

env = StreamExecutionEnvironment.get_execution_environment()
source = env.add_source(KafkaSource(...))
processed = source.map(lambda x: process_event(x))
processed.add_sink(KafkaSink(...))
env.execute("stream_job")

Hadoop (2 года, legacy)

• HDFS: distributed file storage
• MapReduce: старые batch jobs
• Заменён Spark'ом (более быстро и удобно)

Databases & Data Warehouses

PostgreSQL (5+ лет)

• Production OLTP база
• Transactional consistency (ACID)
• Реляционные данные

-- Типичная schema для transactional системы
CREATE TABLE users (
    id BIGINT PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE orders (
    id BIGINT PRIMARY KEY,
    user_id BIGINT REFERENCES users(id),
    amount DECIMAL(10,2),
    status VARCHAR(50),
    created_at TIMESTAMP
);

Snowflake (3+ года)

• Cloud data warehouse
• SQL аналитика
• Масштабируется от GB до EB
• Автоматическое управление инфраструктурой

-- Snowflake витрина
CREATE OR REPLACE TABLE fact_sales AS
SELECT 
    DATE_TRUNC('day', timestamp) as sale_date,
    product_id,
    SUM(amount) as total_sales,
    COUNT(*) as transaction_count
FROM raw.events
WHERE event_type = 'purchase'
GROUP BY 1, 2;

BigQuery (3+ года)

• Google's serverless DW
• Petabyte-scale analytics
• Встроена ML (BigQuery ML)

# BigQuery через Python
from google.cloud import bigquery

client = bigquery.Client()
query = """
    SELECT 
        DATE(purchase_ts) as date,
        COUNT(*) as daily_sales
    FROM `project.dataset.events`
    WHERE purchase_ts >= '2026-03-01'
    GROUP BY date
"""
df = client.query(query).to_dataframe()

Amazon Redshift (2 года)

• AWS data warehouse
• PostgreSQL-compatible
• Columnar storage для аналитики

ClickHouse (1 год, for analytics)

• Экстремально быстрый для аналитики
• Columnar format
• Огромные агрегации за секунды

-- ClickHouse query для 100B+ строк
SELECT 
    year,
    month,
    SUM(amount) as total
FROM sales
GROUP BY year, month;
-- Выполняется за 1-2 секунды, vs Spark 30+ секунд

MongoDB (3 года, legacy)

• Document database
• Flexible schema
• Использовал для user profiles и cache

Redis (6+ лет)

• In-memory cache
• Session storage
• Real-time counters
• Rate limiting

from redis import Redis

redis = Redis(host='localhost')
redis.set('user:123:name', 'John')  # microseconds
redis.incr('daily_active_users')    # atomic increment
redis.expire('session:abc', 3600)   # TTL

Cassandra (2 года)

• Distributed NoSQL
• High write throughput
• Time-series data

DynamoDB (3 года, AWS serverless)

• Managed key-value store
• Auto-scaling
• Good for serverless applications

Data Integration & Orchestration

Apache Airflow (4+ лет)

• DAG-based workflow orchestration
• Scheduling ETL pipelines
• Monitoring и alerting

# Airflow DAG
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'data-eng',
    'retries': 2,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG(
    'daily_sales_etl',
    default_args=default_args,
    schedule_interval='0 2 * * *',  # Every day at 2 AM
    start_date=datetime(2026, 1, 1)
)

def extract_data():
    # Extract from source
    pass

def transform_data():
    # Transform
    pass

def load_data():
    # Load to warehouse
    pass

ext_task = PythonOperator(task_id='extract', python_callable=extract_data, dag=dag)
trf_task = PythonOperator(task_id='transform', python_callable=transform_data, dag=dag)
lod_task = PythonOperator(task_id='load', python_callable=load_data, dag=dag)

ext_task >> trf_task >> lod_task

Prefect (1 год)

• Более modern alternative к Airflow
• Лучше для data flows
• Динамичнее чем DAG'ы

Dagster (1 год, текущий стандарт)

• Production-grade orchestration
• Asset-oriented (не task-oriented)
• Встроена типизация и validation

# Dagster asset
from dagster import asset, In, Out

@asset
def raw_events(context) -> pd.DataFrame:
    """Load raw events from source"""
    return pd.read_csv('events.csv')

@asset
def clean_events(raw_events: pd.DataFrame) -> pd.DataFrame:
    """Clean and validate events"""
    return raw_events[raw_events['amount'] > 0].drop_duplicates()

@asset
def daily_sales(clean_events: pd.DataFrame) -> pd.DataFrame:
    """Aggregate to daily sales"""
    return clean_events.groupby('date').agg({'amount': 'sum'})

Apache NiFi (1 год)

• Visual data routing
• Data provenance
• Хорошо для complex routing logic

dbt (3+ года, essential tool!)

• Transform data in SQL
• Version control for SQL
• Testing и documentation

-- dbt model (models/staging/stg_orders.sql)
{{ config(
    materialized='table',
    indexes=[
        {'columns': ['user_id'], 'unique': False}
    ]
) }}

SELECT 
    o.id as order_id,
    o.user_id,
    o.amount,
    o.created_at,
    u.country,
    u.email
FROM {{ source('raw', 'orders') }} o
JOIN {{ ref('dim_users') }} u ON o.user_id = u.user_id
WHERE o.amount > 0

# dbt tests (tests/stg_orders.yml)
models:
  - name: stg_orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: amount
        tests:
          - not_null
          - dbt_utils.expression_is_true:
              expression: "> 0"

Message Queues & Streaming

Apache Kafka (4+ лет)

• Event streaming platform
• High throughput (millions of events/sec)
• Distributed, resilient

# Kafka Producer
from kafka import KafkaProducer
import json

producer = KafkaProducer(bootstrap_servers=['localhost:9092'])
for event in events:
    producer.send('events-topic', json.dumps(event).encode())

# Kafka Consumer
from kafka import KafkaConsumer

consumer = KafkaConsumer('events-topic', bootstrap_servers=['localhost:9092'])
for msg in consumer:
    event = json.loads(msg.value.decode())
    process(event)

RabbitMQ (1 год)

• Message broker
• Request/reply pattern
• Less scalable than Kafka для big data

AWS SQS (2 года)

• Managed message queue
• Serverless
• Good for decoupling

Cloud Platforms

AWS (5+ лет)

• S3 (object storage)
• EC2 (compute)
• Lambda (serverless)
• EMR (managed Hadoop/Spark)
• Redshift (data warehouse)
• Glue (ETL service)

Google Cloud (4+ лет)

• BigQuery (data warehouse)
• Cloud Storage (object storage)
• Cloud Functions (serverless)
• Dataflow (managed Beam/Spark)
• Cloud Dataproc (managed Hadoop)

Azure (2+ лет)

• Azure Synapse (DW)
• Azure Data Lake Storage
• Databricks (on Azure)
• Azure Functions

Visualization & BI

Tableau (2 года)

• Interactive dashboards
• VizQL for complex queries
• Хорошо для executive dashboards

Power BI (2 года)

• Microsoft BI tool
• Integration with Excel
• Good for enterprise

Looker (3 года, prefer this)

• Встроена в BigQuery
• LookML for defining metrics
• Self-serve analytics

# Looker dashboard
view: orders {
  sql_table_name: public.orders ;;

  dimension: id {
    primary_key: yes
    type: number
    sql: ${TABLE}.id ;;
  }

  measure: total_revenue {
    type: sum
    sql: ${TABLE}.amount ;;
  }

  measure: order_count {
    type: count
  }
}

Metabase (1 год, open-source)

• Self-hosted BI tool
• Simple query builder
• Good for startups

Languages & Frameworks

Python (10+ лет)

• Primary language
• Pandas, NumPy, PySpark
• FastAPI for APIs

# Typical Python script for data processing
import pandas as pd
from sqlalchemy import create_engine

engine = create_engine('postgresql://user:pass@localhost/db')

df = pd.read_sql('SELECT * FROM raw_events', engine)
df_clean = df[df['amount'] > 0].drop_duplicates()
df_clean.to_sql('clean_events', engine, if_exists='replace')

SQL (10+ лет)

• PostgreSQL, Snowflake, BigQuery dialects
• Optimization and tuning

Scala (2 years)

• Spark core development
• Type-safe FP

Java (basics)

• Hadoop ecosystem
• Spring Boot for services

Bash/Scripting (10+ years)

• Automation
• Data pipeline monitoring

Testing & Monitoring

pytest (5+ years)

• Unit testing
• Data validation tests
• Integration tests

def test_clean_amount():
    assert clean_amount(10.556) == 10.56
    assert clean_amount(-5) == 0

def test_data_quality():
    df = load_data()
    assert df.isnull().sum().sum() == 0  # No nulls
    assert df.shape[0] > 0  # Data exists

Great Expectations (2 years)

• Data quality framework
• Automated validation
• Profiling datasets

from great_expectations.dataset import PandasDataset

df = PandasDataset(pd.read_csv('data.csv'))
df.expect_table_row_count_to_be_between(min_value=1000, max_value=100000)
df.expect_column_values_to_be_in_set('country', ['US', 'UK', 'DE'])
df.expect_column_values_to_be_of_type('amount', 'float64')

Prometheus & Grafana (3 years)

• Metrics collection
• Real-time monitoring
• Alerting

DataDog (2 years, commercial)

• Full observability platform
• Logs, metrics, traces
• APM for services

ELK Stack (3 years)

• Elasticsearch, Logstash, Kibana
• Log aggregation
• Search and visualization

Version Control & Deployment

Git (10+ years)

• GitHub, GitLab, Bitbucket
• Feature branches, code review

Docker (6+ years)

• Containerization
• Reproducible environments
• CI/CD pipelines

FROM python:3.9

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "etl.py"]

Kubernetes (basics)

• Orchestration
• Scaling
• Self-healing

CI/CD: GitHub Actions, GitLab CI (4+ years)

• Automated testing
• Deployment pipelines

My Tech Stack (ideal setup)

┌─────────────────────────────────────┐
│ Data Layer                          │
├─────────────────────────────────────┤
│ Data Sources (APIs, Databases)      │
│         ↓                           │
│ Kafka (event streaming)             │
│         ↓                           │
│ Spark (batch + streaming)           │
│         ↓                           │
│ dbt (transformations in SQL)        │
│         ↓                           │
│ Snowflake (data warehouse)          │
│         ↓                           │
│ Looker (analytics & BI)             │
│         ↓                           │
│ Dashboards for business users       │
└─────────────────────────────────────┘

Orchestration: Dagster
Monitoring: Datadog
Testing: pytest + Great Expectations
CI/CD: GitHub Actions

Tools I don't use (and why)

❌ Talend, Informatica — Too enterprise, not flexible enough ❌ Hive — Spark is faster and easier ❌ MapReduce — Spark replaced it long ago ❌ Pig — Why when you have SQL (Hive)? ❌ Tableau — Looker is more modern and integrated

Итог

Я работал с 30+ инструментами. Моя философия:

Use the right tool for the job:

• PostgreSQL для OLTP
• Snowflake/BigQuery для OLAP
• Spark для big data processing
• Kafka для streaming
• dbt для трансформаций
• Dagster для оркестрации
• Looker для analytics

Текущие тренды (2026):

• dbt становится стандартом трансформаций
• Snowflake/BigQuery вытеснили Redshift
• Polars (Rust) заменяет Pandas для больших данных
• Python становится универсальным языком Data Engineering
• Cloud-native architectures (меньше self-hosted инструментов)