16.10.2026 aktualisiert
MG
100 % verfügbar
Data Engineer | Big Data | Azure | PySpark | CI/CD
Skopje, Mazedonien
Nur Remote
BSc. in Information EngineeringSkills
Test AutomationBig DataData ArchitectureData TransformationData WarehousingAzure Data FactorypandasPySparkDaten-PipelineDatabricks
Data Engineering & Pipeline Development
Design and implementation of production-grade analytical data platforms, batch data ingestion pipelines using Azure Data Factory and Databricks, as well as AI-driven data processing pipelines using PySpark. Emphasis on data quality, correctness, and reproducibility.
Python & SQL Development
Development of Python-based libraries, reusable test components, and analytical solutions using PySpark, Pandas, and SQL. Hands-on experience with automated testing frameworks and in-house tooling.
CI/CD & Azure DevOps
Implementation of CI/CD pipelines with Azure DevOps, including automated validation steps for code and data changes. Proficiency with Git, Azure Repos, and Pipelines for reliable software delivery.
Test Automation & Quality Assurance
Conceptualization and implementation of automated test cases for customer-facing software systems, development of test utilities, and support of test engineers in validating devices under test. Contribution to quality assurance processes through tooling and standardization.
Digital Twin & Autonomous Systems
Creation of a Digital Twin for Hardware-in-the-Loop testing of communication systems and models for improvement of autonomous driving capabilities. Involvement in the state-funded European Digital Dynamic Mapping (EDDY) project.
Data Architecture & Modeling
Design of core relational data models for vendor management software and foundational data architectures for multi-agent AI initiatives, utilizing LangChain and vector embeddings to orchestrate complex unstructured data flows.
Databricks & Azure Data Factory
Building and maintaining batch data ingestion pipelines using Databricks and Azure Data Factory, ensuring reliable execution and error handling across complex data environments.
Root-Cause Analysis & Debugging
Support of debugging and root-cause analysis of pipeline failures across environments, as well as systematic validation of analytical outputs through test datasets and checks.
Design and implementation of production-grade analytical data platforms, batch data ingestion pipelines using Azure Data Factory and Databricks, as well as AI-driven data processing pipelines using PySpark. Emphasis on data quality, correctness, and reproducibility.
Python & SQL Development
Development of Python-based libraries, reusable test components, and analytical solutions using PySpark, Pandas, and SQL. Hands-on experience with automated testing frameworks and in-house tooling.
CI/CD & Azure DevOps
Implementation of CI/CD pipelines with Azure DevOps, including automated validation steps for code and data changes. Proficiency with Git, Azure Repos, and Pipelines for reliable software delivery.
Test Automation & Quality Assurance
Conceptualization and implementation of automated test cases for customer-facing software systems, development of test utilities, and support of test engineers in validating devices under test. Contribution to quality assurance processes through tooling and standardization.
Digital Twin & Autonomous Systems
Creation of a Digital Twin for Hardware-in-the-Loop testing of communication systems and models for improvement of autonomous driving capabilities. Involvement in the state-funded European Digital Dynamic Mapping (EDDY) project.
Data Architecture & Modeling
Design of core relational data models for vendor management software and foundational data architectures for multi-agent AI initiatives, utilizing LangChain and vector embeddings to orchestrate complex unstructured data flows.
Databricks & Azure Data Factory
Building and maintaining batch data ingestion pipelines using Databricks and Azure Data Factory, ensuring reliable execution and error handling across complex data environments.
Root-Cause Analysis & Debugging
Support of debugging and root-cause analysis of pipeline failures across environments, as well as systematic validation of analytical outputs through test datasets and checks.
Erfahrung
Berufserfahrung0 - 5 Jahre
Freelancer seit2026
Sprachen
GermanverhandlungssicherEnglishverhandlungssicher
Projekthistorie
Designed the core relational data model for an internal Vendor Management Software. Conceptualized the foundational data architecture for a multi-agent AI initiative using LangChain and vector embeddings. Automated manual documentation workflows by engineering AI-driven data processing pipelines using PySpark.
Designed and implemented a production analytical data platform for a multinational reinsurance client. Built and maintained batch data ingestion pipelines using Azure Data Factory and Databricks. Developed analytical solutions using PySpark, Pandas, and SQL. Implemented CI/CD pipelines with Azure DevOps and supported debugging and root-cause analysis of pipeline failures.
Conceptualized and created a Digital Twin for Hardware-in-the-Loop testing of communication systems and models for improvement of autonomous driving capabilities. Planned implementation and standardization for the state-funded European Digital Dynamic Mapping (EDDY) project. Configured and set up a workstation for neural network training and data processing projects.