AgileEngine is looking for a Senior Big Data Engineer to contribute to the Data Engineering team Marketplace and inventory listings to make car shopping an uplifting experience. The project is to work identifying data to capture Data normalization and adding new features to the existing platform and monitoring features to the pipeline. The ideal candidate’s prior work experience reflects high standards for engineering with proven capabilities.
What is required?
- 2+ years of experience in Big Data technologies (Hadoop (MapReduce), Apache Spark, HBase, AWS Kinesis)
- 5+ years of experience programming in Java
- Experience working with Spring Boot Framework
- Experience working with AWS or any other cloud providers
- Experience working with AWS Kinesis and other AWS ETL tools
- Proficient in SQL and experience with NoSQL databases (HBase, Redis, Elasticsearch)
- Ability to self-manage tasks and be proactive in working with other teams to accomplish them while taking pride and ownership in their work
- Team-player with strong collaboration and communication skills, who is able to respond positively to feedback.
What will be a plus?
- Experience working with Cloudera/Hortonworks/EMR distribution in AWS
What will you do?
- Design and develop efficient and scalable data processing pipelines using big data technologies (Hadoop, Spark, HBase, Kinesis, MapReduce, etc.) on large scale structured/unstructured data sets for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL/NoSQL
- Build complex workflows and orchestrate data dependencies
- Monitor and support data pipelines to honor internal and external SLA’s
- Work within standard engineering practices (i.e. SCRUM, unit/integration testing, design review, code reviews, continuous integration, etc.) to deliver product features with optimal efficiency for customers and clients
- Closely work with product owners & analysts to understand business and functional requirements and contribute to the design and prioritization discussions
- Working with a team of engineers where mentorship is valued
- Ability to learn and adapt to continually evolving technologies in the big data ecosystem
- Interesting and challenging tasks
- Flexible work schedule
- Zero bureaucracy
- Friendly and a very skilled team with great corporate culture and mentorship (visit us and see it yourself)
- US democratic management style
- Opportunities for self-realization, professional and career growth
- Cool events and team activities
- Professional workshops and training, a great engineering culture
About the project?
Our client is a leading automotive digital marketplace that enables car buyers to connect to our nationwide network of Certified Dealers. They are building the industry’s most personalized and efficient car buying experience as we seek to bring more of the purchasing process online. Consumers who visit our marketplace will find a suite of vehicle discovery tools, price ratings, and market context on new and used cars – all with a clear view of what’s a great deal. When they are ready, our client will enable them to connect with a local Certified Dealer who shares in our belief that truth, transparency, and fairness are the foundation of a great car buying experience. As part of our marketplace, our client powers car-buying programs for over 250 leading brands, including AARP, Sam’s Club, and American Express. Nearly half of all new-car buyers engage with our client powered sites, where they buy smarter and drive happier.