How Aampe Handles High-Volume Data Ingestion Without Breaking the Bank

Cost Optimization Strategies

1. Eliminating Data Duplication: Direct Access Pattern

Customer Service Account Pattern: When ingesting from customer GCS buckets, we use customer service accounts to read data directly without copying it to our buckets first:

# Use customer credentials to read from their GCS buckets
customer_credentials = ServiceAccountAuth.get_customer_credentials(
    secret_manager_project, customer_id
)
client = storage.Client(credentials=customer_credentials)

Benefits:

• No data duplication: We read directly from customer buckets, avoiding the need to copy data to Aampe's buckets

• Reduced storage costs: We don't maintain duplicate copies of customer data in our buckets

• Faster processing: Eliminates the intermediate copy step

• Note: While GCS ingress is free, the real savings come from avoiding duplicate storage

Cost Impact:

• Traditional approach: Copy 1TB from customer bucket to Aampe bucket = ~$23/month storage cost for duplicate data

Additional Benefits:

• No ingress operations: While GCS ingress is free, we avoid unnecessary data transfer operations

• Allows to pull only subset of data as noted in next section

2. Minimizing Egress Costs: Incremental Data Pulling

Incremental Extraction: Instead of pulling full tables every day, we use configurable time windows to pull only recent data. This applies to Snowflake, S3, and other data sources where customers still pay egress costs:

# Only pull data from the last N days (Snowflake example)
query = f"""
    SELECT * FROM {table_name}
    WHERE DATE({incremental_by_column}) >= DATEADD(day, -{data_change_window}, DATE('{run_date}'))
    AND DATE({incremental_by_column}) <= DATE('{run_date}')
""

Key Features:

• Configurable windows: data_change_window parameter (typically 0-7 days)

• Partition-aware queries: Only query relevant partitions

• Full load on first run: Initial load pulls everything, subsequent runs are incremental

• Applies to all sources: Snowflake, S3, GCS, and other external data sources

Important Note:

• For Snowflake, S3, and other sources: Customers still pay egress costs, but we minimize them through incremental pulls

• For BigQuery sources: Egress is free (see Billing Project Pattern section below) because BigQuery-to-BigQuery transfers within the same region have no egress charges

Cost Impact for Snowflake/S3 Sources:

• Full table pull: For a 1TB table, egress costs ~$120/month (at $0.12/GB)

• Incremental pull (7 days): For 7 days of data (~50GB), egress costs ~$6/month

Example: A customer with a 500GB events table in Snowflake:

• Daily full pull: 500GB × 30 days = 15TB/month = ~$1,800/month in egress

• Daily incremental (1 day): ~17GB/day × 30 = 510GB/month = ~$61/month

For BigQuery Sources: Egress costs are $0 because BigQuery-to-BigQuery transfers within the same region are free, and we use the billing project pattern (see next section).

3. Billing Project Pattern: Absorbing Query Costs

Aampe as Billing Project: When pulling data from customer BigQuery projects, we use customer credentials but set Aampe's project as the billing project:

# Use customer credentials to access their data
json_credentials = self._get_json_cred_from_secret_manager()
# But bill queries to Aampe's projectreturn bigquery.Client(
    credentials=json_credentials,
    project=settings.BQ_PROJECT_ID# Aampe's project
)

How It Works:

1. We authenticate using customer's service account credentials

2. We query customer's BigQuery tables (with their permission)

3. But the query costs are billed to Aampe's project, not the customer's

Benefits for Customers:

• Zero query costs: Customers don't pay for queries we run on their data

• Zero egress costs: BigQuery-to-BigQuery transfers within the same region are free (unlike Snowflake/S3 where egress costs apply)

• Transparent pricing: Customers only see storage costs, not processing costs

Key Distinction:

• BigQuery sources: $0 egress costs (free BQ-to-BQ transfers) + $0 query costs (billed to Aampe)

• Snowflake/S3 sources: Customers pay egress costs, but we minimize them through incremental pulls (95%+ savings)

Cost Impact: For a customer with daily queries scanning 100GB:

• Customer pays: $0 (queries billed to Aampe)

• Traditional approach: Customer would pay ~$5/day = ~$150/month in query costs

• Customer savings: ~$150/month

Note: This pattern works because:

• We have proper IAM permissions via customer service accounts

• The billing project is just an accounting mechanism - data access is still controlled by customer IAM

4. Strategic BigQuery Partitioning

Daily Time Partitioning: Every table in our system uses daily partitioning, which provides several benefits:

• Query Cost Reduction: Queries can scan only relevant partitions instead of entire tables

• Efficient Data Management: Easy to drop old partitions or backfill specific date ranges

• Better Performance: BigQuery can parallelize queries across partitions

additional_bq_parameters={
    "timePartitioning": {"type": "DAY", "expirationMs": EXPIRATION_MS},
}

Impact: For a table with 180 days of data, a query targeting a single day scans only 0.55% of the data, resulting in ~99.45% cost reduction for date-filtered queries.

5. Intelligent Clustering

We strategically cluster tables on frequently queried fields:

• Event tables: Clustered on event_name, event_type, or user_id

• Attribute tables: Clustered on user_id for fast user lookups

clustering_fields=["event_name", "user_id", "timestamp"]

Impact: Clustering can reduce query costs by 50-90% for queries that filter on clustered columns, as BigQuery can skip entire blocks of data.

6. Automated Data Lifecycle Management

180-Day Expiration Policy: All staging tables automatically expire after 180 days:

EXPIRATION_MS = 180 * 24 * 60 * 60 * 1000# 180 days

Benefits:

• Automatic cleanup: No manual intervention needed

• Cost control: Prevents unbounded storage growth

• Compliance: Helps meet data retention requirements

Cost Impact: For a customer generating 1TB/month, this policy saves ~$20/month in storage costs per customer after the first 180 days, and prevents exponential growth.