Mahmoodi & Strukov "Bring the Noise"
Reza Mahmoodi, a 5-year PhD candidate in the lab of ECE Prof. Dmitri Strukov describes an approach, like the human brain has, to leverage noise (The UCSB Current)
Mohammed “Reza” Mahmoodi, a fifth-year Ph.D. candidate in the lab of ECE Prof. Dmitri Strukov describes an approach to leverage noise, as the human brain does, for better neural networks
Those who design deep neural networks for artificial intelligence often find inspiration in the human brain. One of the brain’s more important characteristics is that it is a “noisy” system: not every neuron contains perfect information that gets carried across a synapse with perfect clarity. Sometimes partial or conflicting information is turned into action by the brain, and sometimes partial information is not acted upon until further information is accumulated over time.
“That is why, when you stimulate the brain with the same input at different times, you get different responses,” explained Mohammed “Reza” Mahmoodi, a fifth-year Ph.D. candidate in the lab of UC Santa Barbara electrical and computer engineering professor Dmitri Strukov. “Noisy, unreliable molecular mechanisms are the reason for getting substantially different neural responses to repeated presentations of identical stimuli, which, in turn, allow for complex stochastic, or unpredictable, behavior.”
The human brain is extremely good at filling in the blanks of missing information and sorting through noise to come up with an accurate result, so that “garbage in” does not necessarily yield “garbage out.” In fact, Mahmoodi said, the brain seems to work best with noisy information. In stochastic computing, noise is used to train neural networks, “regularizing” them to improve their robustness and performance.
It is not clear on what theoretical basis neuronal responses involved in perceptual processes can be separated into a “noise” versus a “signal,” Mahmoodi explained, but the noisy nature of computation in the brain has inspired the development of stochastic neural networks. And those have now become the state-of-the-art approach for solving problems in machine learning, information theory and statistics.
“If you want a stochastic system, you have to generate some noise,” Mahmoodi and his co-authors, Strukov and Mirko Prezioso, write in a paper that describes an approach to creating such a noisy system. “Versatile stochastic dot product circuits based on nonvolatile memories for high performance neurocomputing and neurooptimization” was published in a recent issue of the journal Nature Communications.
