Required Education
•Bachelor's degree in computer science, Information Technology, Engineering, or a related field.
Preferred Education
•Master's degree in computer science, Information Technology, Engineering, or a related field.
Required Qualifications/Skills
•6–8 years of professional software development/data engineering experience.
•Hands-on expertise in PySpark and Big Data processing.
•Experience building and optimizing scalable distributed data pipelines.
•Strong experience with the Hadoop ecosystem, including:
• Hive
• HDFS
• Sqoop
• Spark
• Impala
• Scala
•Strong SQL skills with experience writing complex queries for data extraction, transformation, validation, and analysis.
•Experience designing and developing distributed data systems.
•Solid understanding of distributed systems architecture.
•Experience with data modeling, data design, and data warehouse concepts.
•Experience with dimensional modeling techniques.
•Experience with shell scripting.
•Experience using Autosys or equivalent job scheduling/workflow automation tools.
•Strong analytical and problem-solving skills.
•Ability to independently identify, assess, and resolve technical and data-related issues.
•Strong verbal and written communication skills.
•Ability to communicate technical concepts to both technical and non-technical stakeholders.
Preferred Qualifications/Skills
•Experience with Databricks.
•Experience with BI platforms, preferably Tableau.
•Experience working with real-time and batch streaming data platforms.
Overview
Client is seeking a Senior Big Data Developer / Data Engineer to design, develop, and optimize large-scale data pipelines and distributed data systems that support critical business intelligence initiatives. In this hybrid role based in Mississauga ON, you will work with a high-performing engineering team to build scalable, high-performance data solutions using PySpark, the Hadoop ecosystem, and streaming technologies. The role focuses on designing robust data architectures, developing real-time and batch processing pipelines, optimizing large-scale data workflows, and ensuring high availability of enterprise data platforms. The ideal candidate will have strong experience in distributed systems, SQL, data modeling, workflow automation, and Big Data technologies, along with excellent analytical, problem-solving, and communication skills.
Job Duties
•Build and maintain scalable data pipelines using PySpark to process structured and unstructured data.
•Design and develop solutions using the Hadoop ecosystem, including Hive, HDFS, Sqoop, Spark, Impala, and Scala.
•Develop and manage real-time and batch data processing workflows.
•Ensure high availability and low-latency data delivery across distributed systems.
•Write complex SQL queries for data extraction, validation, transformation, and analysis.
•Design and implement scalable data models aligned with data warehouse principles.
•Develop data architecture patterns to ensure data consistency and scalability.
•Automate pipeline scheduling and orchestration using shell scripting and Autosys.
•Monitor, troubleshoot, and resolve technical and data-related issues.
•Identify and mitigate technical risks to maintain platform stability.
•Collaborate with engineering teams to improve the reliability and performance of enterprise data infrastructure.
- **Only those lawfully authorized to work in the designated country associated with the position will be considered.**
- **Please note that all Position start dates and duration are estimates and may be reduced or lengthened based upon a client’s business needs and requirements.**
Rose International has been great to me. I thank everyone there for all of their hard work; it has not gone unnoticed.
Melody, Consultant
Working for Rose International was the most pleasant assignment I have ever had. They were always on top of situations when necessary, and very helpful. I was very proud to be an employee of Rose International, and would recommend anyone to try to work with them.
Melvon, Consultant
I am very happy with the Rose International, and the professionalism of the employees.
Robin, Consultant
It was great working for Rose International. Everyone was extremely helpful.
Rosann, Consultant
Rose International was not only attentive and responsive, but they were very professional and helpful whenever I called or needed any assistance.
Diane, Consultant
EMPLOYEE COMMENTS