Lessons from Neuroscience for the Machine Learning Community in 2018
The field of machine learning (ML) is rapidly evolving, but it is not without common misconceptions. As of early 2018, several key aspects of the ML community were still plagued by outdated beliefs and practices. The purpose of this article is to provide insights from the recent meeting of the European Institute of Theoretical Neuroscience, which highlighted important lessons for the ML community.
The Technical Challenges and Realities in Machine Learning
Machine learning is often perceived as a mathematical endeavor, but this misconception is far from the truth. In reality, it is more about programming and data handling. Andrew Ng, a prominent figure in the field, echoed this sentiment: The model and the code for many applications are basically a solved problem. Now that the models have advanced to a certain point, we got to make the data work as well.
The majority of the job involves data sourcing and data cleansing, which is a more tedious process than model building. For structured data, classification and regression problems are frequently tackled with gradient boosting techniques such as XGBoost. As a machine learning engineer, this knowledge is invaluable and can help you stand out in the job market.
Biased Perspectives: From Paris to San Francisco
The author reflects on a recent meeting in Paris about Machine Intelligence Meets Neuroscience: What Can MI Learn from Neuroscience. Three key lessons emerged from the discussions:
1. Feedforward Limitations in Machine Learning
Neural networks in biology often use feedback mechanisms, which are crucial for interpreting data. This feedback provides a more comprehensive understanding of the context. Multi-input information theory could offer new insights into how feedback mechanisms work in biological systems.
2. Determinism and Algorithmic Computing
Biology defies determinism, as nothing in it is truly deterministic. Algorithmic computing, which underpins much of current machine learning, is thus limited when applied to real-world scenarios. Biological systems, on the other hand, use complex, non-deterministic processes that are not well-captured by current models.
3. Learning and Development in Biological Systems
Biological learning systems evolve over both evolutionary and developmental timescales. This means that systems designed with fixed architectures may not be suitable for all situations, particularly those that require adaptability like animals facing unforeseeable circumstances. Machine learning models might benefit from more flexible, development-oriented approaches.
4. The Utility of Spiking Neurons
Biological neural networks often use spiking neurons for efficient information coding. This can help reduce power consumption, which is particularly important for silicon-based systems. Spiking neurons also offer a more rapid and efficient way of coding information.
5. Locality of Learning Rules
Modern versions of back-propagation (BP) are non-local, whereas early versions were local. The biological system might be able to leverage faster, local learning rules combined with slower, non-local rules for tasks that require a balance between speed and accuracy.
6. Neuromorphic Systems and Memristor Technology
Neuromorphic systems are designed to mimic the structure and function of biological neurons. There are two types of neuromorphic systems: Spinnaker, a parallel digital computer, and FACETS-based analogue neural network systems. These systems can provide new design spaces for machine learning, although implementing new neuromorphic chips remains challenging.
To conclude, the insights from the meeting indicate that there are many possibilities for machine learning to learn from neuroscience. However, it is challenging to predict which will be most useful in the long term. The future of autonomous thinking machines from the blend of machine learning and neuroscience remains a distant dream, but it is an exciting area with immense potential.
For those serious about learning machine learning, the job prospects and salaries are promising. However, the barrier to entry is high, and only a determined few will succeed. This article aims to inspire and inform those in the ML community, providing a realistic yet optimistic view of the field's future.