DEGCP E-Commerce Data Platform
Production-grade pipeline aggregating product data from Amazon, Noon & Jumia — streaming through Pub/Sub, Dataflow, and BigQuery with dbt incremental transforms.
PythonGCPBigQuerydbtAirflowDocker
View details
Data Engineer · Cairo → the internet
Ahmed Sallam
I turn scattered signals into trustworthy systems.
Scroll to begin
Stage 01 · Ingest
Pull from where it lives.
APIs, event streams, databases, file drops. Land it raw, fast, and append-only.
KafkaKafkaKinesisKinesisS3S3AirbyteAirbyte
Stage 02 · Validate
Enforce shape.
Schema checks, type enforcement, contract tests. Bad data never reaches the lake.
Great ExpectationsGreat ExpectationsdbtdbtJSON SchemaJSON SchemaProtobufProtobuf
Stage 03 · Quality
Watch the watchmen.
Nulls, duplicates, drift, freshness, completeness. Alert before the dashboard lies.
SodaSodaMonte CarloMonte CarlodbtdbtElementaryElementary
Stage 04 · Transform
Clean, enrich, aggregate, model.
Star schemas, slowly changing dims, semantic layers. The raw becomes the model.
dbtdbtSparkSparkSnowflakeSnowflakeBigQueryBigQueryDatabricksDatabricks
Stage 05 · Orchestrate
Schedule, depend, retry, observe.
Idempotent DAGs, backfills that don't break prod, alerts that wake the right person.
AirflowAirflowDagsterDagsterPrefectPrefectArgoArgo
Stage 06 · Serve Analytics
Hand it to the people who ask questions.
Warehouses, lakehouses, semantic models, BI tools. The system becomes a product. From chaos to queryable truth.
SnowflakeSnowflakeBigQueryBigQueryLookerLookerHexHexModeMode
Active sessions · last 24h
- Users
- 25%
- Events
- 20%
- Transactions
- 18%
- Sessions
- 15%
- Referrals
- 12%
- Other
- 10%
Selected Work
Systems that made the number trustable.
01 / 06
Ground truth
Principles I build by.
ACID
PO
MY
SP
CO
Transactions are atomic, consistent, isolated, durable — your data never half-updates. Pick the right isolation level or pay in race conditions.
PostgreSQL · MySQL · Spanner · CockroachDB
CAP
CA
DY
CD
MO
In a distributed system, you pick two of: consistency, availability, partition tolerance. When the network splits, what breaks first?
Cassandra · DynamoDB · Cosmos DB · MongoDB
Idempotency
AI
DB
SP
Running the same job twice produces the same result. The only reason backfills don't end careers.
Airflow · dbt · Spark
Schema Evolution
AV
PR
JS
SR
Your data shape will change. Backward- and forward-compatible schemas mean nothing breaks — not producers, not consumers, not the marts.
Avro · Protobuf · JSON Schema · Schema Registry
Data Lineage
OP
DA
DB
AI
If you can't trace a dashboard number back to its source row, you don't have a pipeline — you have a hope.
OpenLineage · DataHub · dbt · Airflow
Observability
MC
SO
EL
DA
Logs, metrics, lineage, freshness, volume. You can't love what you can't measure. Alert on the symptom, not the cause.
Monte Carlo · Soda · Elementary · Datadog