DEGCP E-Commerce Data Platform
Production-grade pipeline aggregating product data from Amazon, Noon & Jumia — streaming through Pub/Sub, Dataflow, and BigQuery with dbt incremental transforms.
PythonGCPBigQuerydbtAirflow+1
View details
Data Engineer · Cairo
Ahmed Sallam
I turn scattered signals into trustworthy systems.
Scroll to begin
Stage 01 · Ingest
Pull from where it lives.
APIs, event streams, databases, file drops. Land it raw, fast, and append-only.
KafkaKafkaFlinkFlinkPythonPythonAWSAWS
Stage 02 · Validate
Enforce shape.
Schema checks, type enforcement, contract tests. Bad data never reaches the lake.
SQLSQLPythonPythondbtdbt
Stage 03 · Quality
Watch the watchmen.
Nulls, duplicates, drift, freshness, completeness. Alert before the dashboard lies.
dbtdbtAirflowAirflowSQLSQL
Stage 04 · Transform
Clean, enrich, aggregate, model.
Star schemas, slowly changing dims, semantic layers. The raw becomes the model.
SparkSparkdbtdbtSQLSQLHadoopHadoop
Stage 05 · Orchestrate
Schedule, depend, retry, observe.
Idempotent DAGs, backfills that don't break prod, alerts that wake the right person.
AirflowAirflowDockerDockerGitGit
Stage 06 · Serve Analytics
Hand it to the people who ask questions.
Warehouses, lakehouses, semantic models, BI tools. The system becomes a product. From chaos to queryable truth.
Power BIPower BIHiveHiveSQL ServerSQL Server
Active sessions · last 24h
- Users
- 25%
- Events
- 20%
- Transactions
- 18%
- Sessions
- 15%
- Referrals
- 12%
- Other
- 10%
Selected Work
Systems that made the number trustable.
01 / 06
Toolbelt
Skills I ship with.
Core tools and concepts I use to move data — with an honest read on how deep each one goes.
Python90%
SQL92%
ETL / ELT90%
Data Warehousing88%
Data Modeling85%
Batch & Streaming Pipelines85%
SQL Server88%
Apache Airflow85%
Apache Kafka82%
Apache Spark80%
Apache Flink70%
Hadoop73%
Hive70%
Google Cloud Platform82%
AWS72%
Git80%
Docker82%
dbt78%
Power BI80%
Ground truth
Principles I build by.
ACID
Transactions are atomic, consistent, isolated, durable — your data never half-updates. Pick the right isolation level or pay in race conditions.
PostgreSQL · MySQL · Spanner · CockroachDB
CAP
In a distributed system, you pick two of: consistency, availability, partition tolerance. When the network splits, what breaks first?
Cassandra · DynamoDB · Cosmos DB · MongoDB
Idempotency
Running the same job twice produces the same result. The only reason backfills don't end careers.
Airflow · dbt · Spark
Schema Evolution
Your data shape will change. Backward- and forward-compatible schemas mean nothing breaks — not producers, not consumers, not the marts.
Avro · Protobuf · JSON Schema · Schema Registry
Data Lineage
If you can't trace a dashboard number back to its source row, you don't have a pipeline — you have a hope.
OpenLineage · DataHub · dbt · Airflow
Observability
Logs, metrics, lineage, freshness, volume. You can't love what you can't measure. Alert on the symptom, not the cause.
Monte Carlo · Soda · Elementary · Datadog