Navigating the Challenges of Machine Learning Research
Written on
Chapter 1: The Allure and Reality of ML Research
Machine Learning (ML) is undeniably one of the most exciting fields today. If you find yourself working as a researcher in a reputable lab, start-up, or major tech company, you’re at the forefront of what could be one of humanity's most transformative technologies, often accompanied by a lucrative salary.
However, while I don’t wish to diminish the appeal of an ML researcher’s career, it's essential to highlight the challenges one might encounter in this field. Having spent three years as an ML student researcher, collaborating with a Google DeepMind researcher, and publishing a paper at a prestigious conference—albeit with two rejections—I can offer insights into these realities.
Let's dive into the intricacies of pursuing a career in ML research.
Section 1.1: Job Prospects and Requirements
So, how feasible is it for you to become an ML researcher or engineer earning six figures at a leading firm like OpenAI, Google, or Meta? The blunt truth is: it’s quite challenging, though not unattainable.
Factors such as your experience, enthusiasm, networking, and, notably, luck, all play a role. A common question among aspiring ML researchers is whether a Ph.D. is necessary for obtaining a position at a top company. The straightforward answer is: not necessarily, but it can be very beneficial. Alternatively, relevant experience may suffice, which I can elaborate on if there’s interest.
While a Ph.D. is not mandatory for most ML engineering roles, it often serves as a prerequisite for research positions. This is logical, as research roles demand domain-specific expertise, typically acquired through years of dedicated study on a particular issue. A Ph.D. provides the invaluable opportunity to spend 3-5 years honing in on a specific research topic.
In the hierarchy of roles, a machine learning researcher, or research scientist, generally holds a higher position and earns more than an ML engineer. The latter typically assists in implementing the researcher's concepts, particularly in larger companies that differentiate between these roles.
However, possessing a Ph.D. doesn’t guarantee success. I know several Ph.D. holders whose skills vary significantly. Some are exceptional, while others are merely average or below par. Proving your worth through publications, collaborations, and clear communication is vital, and this can be a harsh reality for many.
For instance, I’ve encountered Ph.D. holders who are incredibly talented yet humble, enjoying their research journey without the pressures of being overly ambitious. They thrive in their roles at esteemed institutions like Google, driven by genuine passion for their work.
Ultimately, you must have a deep love for your field to sustain motivation over years without immediate financial reward, or you need to discover this passion early in your academic journey, showcasing outstanding work through personal projects, open-source contributions, or leading publications.
The path to becoming a successful ML researcher is not a quick one; it often requires several years of dedication, and a fair amount of luck plays into the mix.
Section 1.2: The Importance of Coding Skills
Let’s shift our focus to a crucial skill in ML research: coding. In earlier times, physics research often involved small teams generating theories with basic equipment. Fast forward to today, and to contribute meaningfully to modern physics, one must often work at a facility like CERN, equipped with advanced technology.
Similarly, ML research has evolved. A few years ago, developing ideas could be accomplished with relatively simple coding. A landmark development in deep learning, ResNet, exemplifies this; the code modification was minimal yet impactful.
However, with the rise of large models, researchers now need extensive knowledge in distributed computing, hardware optimization, and managing large-scale infrastructures. This is why many big firms delineate the roles of research scientists and ML engineers.
Fortunately, there are numerous ML domains that don’t necessitate working with massive models. Although discussing each would be extensive, I have a comprehensive list of these domains available for free.
Depending on your preferences, the depth of software knowledge required in ML research may excite you. However, if you’re inclined towards research to avoid engineering tasks, you may need to reconsider your focus.
Chapter 2: The Realities of Conducting Research
If you've resolved to tackle the challenges of pursuing a master’s or Ph.D., what can you expect from the research journey? In my experience, there’s a captivating yet frustrating aspect to research: you are often working on something groundbreaking, ideally something that hasn’t been explored before.
Beginning your Ph.D. can be daunting; determining your research focus can take time. For instance, my area of interest is video-language modeling, which encompasses numerous unresolved issues.
Initial excitement can quickly turn into frustration after months of effort yield no substantial results. You may face technical setbacks, theoretical hurdles, or realize that your initial hypothesis is flawed or irrelevant. This uncertainty can lead to anxiety about the validity of your research.
Despite these challenges, it’s crucial to trust the process. Remember that negative outcomes can still provide valuable insights for refining your research question and design. Seeking feedback from mentors and peers and delving deeper into literature can open new avenues for exploration.
Eventually, you may discover positive results that bolster your hypothesis or yield intriguing findings, leading you to draft a paper for submission. However, the submission process can be nerve-wracking, especially with the unpredictable nature of peer reviews.
I recently experienced this firsthand when submitting a paper. While one reviewer provided constructive feedback, another offered a disheartening evaluation without substantial critique. This inconsistency in reviews highlights the subjectivity of the process.
A 2021 study at the NeurIPS conference revealed that 50% of accepted papers by one committee would have been rejected by another, emphasizing that the review process doesn’t always reflect the true quality of your work.
After reassessing feedback and improving our paper, we resubmitted it to another conference, waiting for the reviews with a sense of camaraderie in this shared experience.
In conclusion, the journey to becoming a top-tier machine learning researcher is fraught with challenges, as I’ve experienced so far. I’m certain there are many more facets to explore, but before you embark on this path, consider avoiding common pitfalls. You might want to check out my next post, where I outline seven mistakes you could be making already!
Thank you for reading. Farewell!