Job Information
Insight Global Sr Data Engineer in Englewood Cliffs, New Jersey
Job Description
A Large Media Client is seeking a Data Engineer with strong Python and AWS technical expertise to join their Data Engineering team. The Data Engineer will play a crucial role in designing, implementing, and maintaining scalable data pipelines and solutions on the AWS platform. Your work will involve handling large volumes of diverse data, ensuring its accuracy, and providing real-time streaming pipelines and batch processing. The day-to-day will involve building and optimizing pipelines for processing terabytes of data on a varying increments, including monthly, weekly, daily, hourly. This includes both batch processing (daily schedules, hourly updates) and real-time streaming pipelines. The Data Engineer will work closely with cross-functional teams (Cyber, Cloud Ops, AWS, Databricks, partners, and vendors) to address technical issues and resolutions. The Data Engineer will utilize Terraform to write infrastructure code and AWS services, including but not limited to Lambda, SQS, SNS, S3, and Airflow.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com .
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .
Skills and Requirements
- 5-8+ years of experience as a Data Engineer
- Python and PySpark experience at the programming (OOP) level
- Experience with Kafka for batch processing and/or streaming pipelines
o Relative experience with GCP Dataproc, GCP Pub/Sub, RabbitMQ, AWS Redis, or Spark Streaming is okay
- Proficiency with AWS utilizing Lambda, S3, SQS, SNS, and Kinesis
- AWS EMR or Databricks experience - Airflow and DAG -- AWS Services
- AWS Certification
- Lead experience and ability to mentor junior developers null
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to HR@insightglobal.com.