Section 1.1: Lesson 1 - The Importance of Soft Skills

For a long time, I doubted my abilities as an engineer, believing I was "not technical enough." I struggled with algorithm interviews and never contributed to open-source projects, preferring GitHub Desktop over command-line Git. Eventually, I realized I was holding myself back.

As I advanced in my career, I found that my time was increasingly spent collaborating with partners and fellow engineers rather than just coding. To address data challenges effectively, it’s essential to grasp the broader business context and engage in design and iteration. These preparatory steps necessitate strong soft skills, which can sometimes outweigh technical expertise.

Key soft skills that I have identified as vital in my day-to-day work include:

• Communication: Can you articulate engineering challenges to non-engineers? Are you able to create clear documentation and engage in productive team discussions via email or Slack?
• Teamwork: Collaboration takes many forms, from integrating with diverse systems to running effective cross-functional meetings and providing constructive feedback.
• Navigating Ambiguity: Many data problems lack clear definitions, requiring you to gather requirements and refine the scope before devising a solution.
• Ownership: This involves not just completing tasks but ensuring the quality of your work. Data can obscure issues, and it's vital to understand the source data and validate output quality.
• Saying “No”: Good data engineers push back against misguided requests, advocating for better solutions.
• Empathy: Understanding different perspectives enhances support for your colleagues, leading to more informed decisions.
• Public Speaking: While related to communication, this skill is crucial for visibility and personal brand building.

Every data engineer has unique strengths, akin to superpowers in the X-Men universe. Identifying and leveraging your superpowers can significantly enhance your career trajectory.

Section 1.2: Lesson 2 - The Power of Networking

During my first week at Target, my manager Brindha sent me an email listing over ten colleagues with whom I should have coffee. As an introvert, the thought was daunting, yet I pushed myself to meet everyone. Initially awkward, I gradually became comfortable and even arranged additional meetings outside the provided list.

While not every workplace fosters a "coffee chat" culture, I soon realized the importance of these connections. Building relationships with coworkers enhances trust, fosters a sense of belonging, and facilitates diverse perspectives. Furthermore, networking outside your organization can open doors to opportunities that are often accessible through reputation and referrals.

For example, I likely wouldn’t have landed an interview at Netflix if I hadn’t attended the Women in Data meetup in 2019, where I connected with several data engineering managers. Keeping in touch via LinkedIn eventually led to my hiring.

Networking requires time and effort beyond your usual work hours, but it’s a valuable investment in your long-term career.

Video: Data Engineering Was Hard Until I Learned These 5 Secrets!

Section 1.3: Lesson 3 - Cultivating Curiosity and Passion

When asked why I chose data engineering, my response is simple: “Every data problem is like a puzzle, and I love solving puzzles!” The process of sorting through countless data pieces to uncover insights mirrors the satisfaction of completing a jigsaw puzzle.

However, the reality of data engineering is not always sunny. The rapid evolution of big data technologies necessitates continuous learning. The exploration and validation of data often involve numerous iterations. If you lack curiosity or passion for data, crafting effective solutions becomes increasingly challenging. Although these qualities might seem cliché, they are critical traits sought after in data engineer candidates.

When you genuinely enjoy tackling data challenges, you’ll find fulfillment in your daily tasks. Passion can help sustain you through the inevitable ups and downs of your career.

Section 1.4: Lesson 4 - Recognizing When to Move On

A Chinese proverb states that a diamond shines anywhere. This led me to believe that as a skilled engineer, I could thrive in any role. However, not every position or team is the right fit, and it’s essential to acknowledge this fact.

I often blamed myself when things went awry at work. For instance, during periods of burnout, I would criticize myself for not being productive without considering my manager's expectations. While self-reflection is important, external factors such as company culture and management competence also play significant roles in shaping your work experience.

Many hesitate to leave unsatisfying positions due to concerns about job-hopping. However, during interviews, the question of your previous job’s departure will arise regardless of tenure, providing an opportunity to clarify what you seek in future roles. Job hopping can be a smart strategy, particularly in an environment where companies frequently offer better compensation to new hires.

If you feel trapped in your current position, know that there are better opportunities out there.

Section 1.5: Lesson 5 - Prioritizing Self-Care

Throughout my career, I’ve grappled with the persistent feeling of inadequacy. Growing up in China, the emphasis was always on hard work and improvement, often without consideration for my mental health. I chased one goal after another, believing that happiness lay in securing a full-time job or landing my dream role. However, these achievements did not bring the fulfillment I sought.

It wasn’t until I started therapy that I began to find peace with my thoughts. I learned that I deserve happiness and respect, irrespective of my career milestones. Mental health challenges are prevalent in the tech industry, yet they often remain unspoken. Many of my fellow data engineers also experience feelings of anxiety or imposter syndrome.

If you’re facing similar struggles, remember that you’re not alone, and there are professionals who can help you find balance and joy.

Building a career is akin to a train journey, unique for each individual. The ride may be turbulent, and you might question your choices along the way. Regardless of the destination, remember to appreciate the journey.

What are your thoughts? What lessons do you wish you had learned earlier in your career? Share your insights in the comments!

See you next time!

For more insights into Data Engineering, follow Data Engineer Things for additional articles!

The Ultimate Guide to Data Engineer Interviews

What to expect and how to prepare for data engineering interviews.

Data Engineering Excellence at Netflix

Insights from two years as a data engineer at Netflix.

Introduction to Data Engineering

A Q&A addressing frequently asked questions about data engineering.