{"canonical_url":"https://engineer.company/portfolio/designed-and-ran-6-etl-elt-pipelines/","content_markdown":"# Designed and ran 6 ETL/ELT pipelines\n\nDesigned, implemented, and administered 6 ETL/ELT pipelines, utilizing Google BigQuery, MSSQL, PostgreSQL, Shell scripting, PL/pgSQL, and Transact‑SQL, integrating data for efficient Python API processing.\n\n2022 – 2024\n\n**Situation.** The organization needed to consolidate and process large volumes of tabular data from a remote data warehouse located behind an IPsec VPN. This data was crucial for powering internal analytics dashboards and external Python APIs used by clients and partners.\n\n**Task.** The objective was to design, implement, and maintain a set of robust and automated ETL/ELT pipelines to securely retrieve, transform, and load data into Google BigQuery, ensuring data accuracy, performance, and scalability across systems.\n\n**Action.** 6 end‑to‑end ETL/ELT pipelines were designed, implemented, and administered, spanning multiple technologies including Google BigQuery, MSSQL, PostgreSQL, Shell scripting, PL/pgSQL, and Transact‑SQL.\n\n- Established secure connections to a remote server protected by an IPsec VPN to automate data retrieval.\n- Scheduled and executed downloads of compressed data archives containing Parquet, CSV, and .bak files.\n- Developed Bash and Python scripts to extract and classify files based on type and schema.\n- For .bak files, backups were restored into a local MSSQL Server instance using RESTORE DATABASE workflows and schema integrity validated.\n- Loaded tabular data into staging schemas in MSSQL and PostgreSQL, using bcp, psql, and SSIS tools, depending on the source format.\n- Wrote modular and reusable PL/pgSQL and T‑SQL procedures to clean, normalize, and enrich the data based on business logic.\n- Exported curated datasets and tables into intermediate formats, compressed them using gzip, and securely transferred them to a Google Cloud Storage bucket.\n- Automated the upload and schema mapping processes to Google BigQuery, using bq CLI and Python‑based data ingestion scripts.\n\n**Result.** These automated pipelines significantly reduced the manual overhead and processing time — from 3+ hours of manual work to under 20 minutes end‑to‑end, while improving data freshness from weekly to daily syncs. The Python APIs consuming this data saw a 30% performance improvement, and the enhanced visibility helped business analysts deliver faster insights to stakeholders. The solution remains scalable and extensible for onboarding new data sources as the business grows.\n\n---\n\n- Role: Data Engineer\n- Categories: [Databases](https://software.engineer.company/categories/databases/), [PostgreSQL](https://software.engineer.company/categories/postgresql/), [SQL](https://software.engineer.company/categories/sql/), [Data Warehousing](https://software.engineer.company/categories/data-warehousing/), [Data Engineering](https://software.engineer.company/categories/data-engineering/), [Data Pipelines (ETL/ELT)](https://software.engineer.company/categories/data-pipelines/), [Cloud](https://software.engineer.company/categories/cloud/), [Automation \u0026 CI/CD](https://software.engineer.company/categories/automation/), [Networking \u0026 VPN](https://software.engineer.company/categories/networking/), [Python](https://software.engineer.company/categories/python/)\n- Services: [Backend \u0026 API Development](https://software.engineer.company/services/backend-development/), [Database Design \u0026 Modeling](https://software.engineer.company/services/database-design/), [Data Pipeline Development (ETL/ELT)](https://software.engineer.company/services/data-pipeline-development/), [Data Warehouse Design](https://software.engineer.company/services/data-warehousing/), [AI Data Readiness](https://software.engineer.company/services/ai-data/)\n\n\u003chttps://engineer.company/portfolio/designed-and-ran-6-etl-elt-pipelines/\u003e\n","date_modified":"2026-10-21T16:38:05Z","description":"Designed and ran 6 ETL/ELT pipelines (BigQuery, MSSQL, PostgreSQL), cutting a 3-hour manual job to under 20 minutes with daily syncs.","formats":{"gemtext":"https://software.engineer.company/portfolio/designed-and-ran-6-etl-elt-pipelines/index.gmi","gopher":"https://software.engineer.company/portfolio/designed-and-ran-6-etl-elt-pipelines/gophermap","html":"https://software.engineer.company/portfolio/designed-and-ran-6-etl-elt-pipelines/","json":"https://software.engineer.company/portfolio/designed-and-ran-6-etl-elt-pipelines/index.json","markdown":"https://software.engineer.company/portfolio/designed-and-ran-6-etl-elt-pipelines/index.md","pdf":"https://software.engineer.company/portfolio/designed-and-ran-6-etl-elt-pipelines/index.pdf"},"kind":"page","language":"en","title":"Designed and ran 6 ETL/ELT pipelines","url":"https://software.engineer.company/portfolio/designed-and-ran-6-etl-elt-pipelines/","version":1}