What is Neural-Symbolic Integration? by Gustav Šír

”, the answer will be that an apple is “a fruit,” “has red, yellow, or green color,” or “has a roundish shape.” These descriptions are symbolic because we utilize symbols (color, shape, kind) to describe an apple. While the aforementioned correspondence between the propositional logic formulae and neural networks has been very direct, transferring the same principle to the relational setting was a major challenge NSI researchers have been traditionally struggling with. The issue is that in the propositional setting, only the (binary) values of the existing input propositions are changing, with the structure of the logical program being fixed. However, as imagined by Bengio, such a direct neural-symbolic correspondence was insurmountably limited to the aforementioned propositional logic setting. Lacking the ability to model complex real-life problems involving abstract knowledge with relational logic representations (explained in our previous article), the research in propositional neural-symbolic integration remained a small niche.

In the CLEVR challenge, artificial intelligences were faced with a world containing geometric objects of various sizes, shapes, colors and materials. The AIs were then given English-language questions (examples shown) about the objects in their world. Armed with its knowledge base and propositions, symbolic AI employs an inference engine, which uses rules of logic to answer queries. Asked if the sphere and cube are similar, it will answer “No” (because they are not of the same size or color). The next step for us is to tackle successively more difficult question-answering tasks, for example those that test complex temporal reasoning and handling of incompleteness and inconsistencies in knowledge bases. The AMR is aligned to the terms used in the knowledge graph using entity linking and relation linking modules and is then transformed to a logic representation.5 This logic representation is submitted to the LNN.

Symbolic AI

Integrating this form of cognitive reasoning within deep neural networks creates what researchers are calling neuro-symbolic AI, which will learn and mature using the same basic rules-oriented framework that we do. The second module uses something called a recurrent neural network, another type of deep net designed to uncover patterns in inputs that come sequentially. (Speech is sequential information, for example, and speech recognition programs like Apple’s Siri use a recurrent network.) In this case, the network takes a question and transforms it into a query in the form of a symbolic program. The output of the recurrent network is also used to decide on which convolutional networks are tasked to look over the image and in what order.

To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analog to the human concept learning, given the parsed program, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide searching over the large compositional space of images and language.

From Logic to Deep Learning

This attribute makes it effective at tackling problems where logical rules are exceptionally complex, numerous, and ultimately impractical to code, like deciding how a single pixel in an image should be labeled. “Neuro-symbolic [AI] models will allow us to build AI systems that capture compositionality, causality, and complex correlations,” Lake said. This approach was experimentally verified for a few-shot image classification task involving a dataset of 100 classes of images with just five training examples per class.

2) The two problems may overlap, and solving one could lead to solving the other, since a concept that helps explain a model will also help it recognize certain patterns in data using fewer examples. Deep learning has its discontents, and many of them look to other branches of AI when they hope for the future. Such transformed binary high-dimensional vectors are stored in a computational memory unit, comprising a crossbar array of memristive devices. A single nanoscale memristive device is used to represent each component of the high-dimensional vector that leads to a very high-density memory. The similarity search on these wide vectors can be efficiently computed by exploiting physical laws such as Ohm’s law and Kirchhoff’s current summation law.

Although operating with 256,000 noisy nanoscale phase-change memristive devices, there was just a 2.7 percent accuracy drop compared to the conventional software realizations in high precision. These soft reads and writes form a bottleneck when implemented in the conventional von Neumann architectures (e.g., CPUs and GPUs), especially symbolic ai for AI models demanding over millions of memory entries. Thanks to the high-dimensional geometry of our resulting vectors, their real-valued components can be approximated by binary, or bipolar components, taking up less storage. More importantly, this opens the door for efficient realization using analog in-memory computing.

If such an approach is to be successful in producing human-like intelligence then it is necessary to translate often implicit or procedural knowledge possessed by humans into an explicit form using symbols and rules for their manipulation. Artificial systems mimicking human expertise such as Expert Systems are emerging in a variety of fields that constitute narrow but deep knowledge domains. While the particular techniques in symbolic AI varied greatly, the field was largely based on mathematical logic, which was seen as the proper (“neat”) representation formalism for most of the underlying concepts of symbol manipulation. With this formalism in mind, people used to design large knowledge bases, expert and production rule systems, and specialized programming languages for AI.

Code, Data and Media Associated with this Article

This idea has also been later extended by providing corresponding algorithms for symbolic knowledge extraction back from the learned network, completing what is known in the NSI community as the “neural-symbolic learning cycle”. However, there have also been some major disadvantages including computational complexity, inability to capture real-world noisy problems, numerical values, and uncertainty. Due to these problems, most of the symbolic AI approaches remained in their elegant theoretical forms, and never really saw any larger practical adoption in applications (as compared to what we see today). We hope this work also inspires a next generation of thinking and capabilities in AI. AI researchers like Gary Marcus have argued that these systems struggle with answering questions like, “Which direction is a nail going into the floor pointing?” This is not the kind of question that is likely to be written down, since it is common sense.

• This approach was experimentally verified for a few-shot image classification task involving a dataset of 100 classes of images with just five training examples per class.
• But it can be challenging to reuse these deep learning models or extend them to new domains.
• Knowledge-based systems have an explicit knowledge base, typically of rules, to enhance reusability across domains by separating procedural code and domain knowledge.
• It wasn’t until the 1980’s, when the chain rule for differentiation of nested functions was introduced as the backpropagation method to calculate gradients in such neural networks which, in turn, could be trained by gradient descent methods.
• Neurosymbolic AI is also demonstrating the ability to ask questions, an important aspect of human learning.

The true resurgence of neural networks then started by their rapid empirical success in increasing accuracy on speech recognition tasks in 2010 [2], launching what is now mostly recognized as the modern deep learning era. Shortly afterward, neural networks started to demonstrate the same success in computer vision, too. Historically, the two encompassing streams of symbolic and sub-symbolic stances to AI evolved in a largely separate manner, with each camp focusing on selected narrow problems of their own. Originally, researchers favored the discrete, symbolic approaches towards AI, targeting problems ranging from knowledge representation, reasoning, and planning to automated theorem proving. Such causal and counterfactual reasoning about things that are changing with time is extremely difficult for today’s deep neural networks, which mainly excel at discovering static patterns in data, Kohli says.

Problems with Relational-Level NSI

Symbols also serve to transfer learning in another sense, not from one human to another, but from one situation to another, over the course of a single individual’s life. That is, a symbol offers a level of abstraction above the concrete and granular details of our sensory experience, an abstraction that allows us to transfer what we’ve learned in one place to a problem we may encounter somewhere else. In a certain sense, every abstract category, like chair, asserts an analogy between all the disparate objects called chairs, and we transfer our knowledge about one chair to another with the help of the symbol. Another way the two AI paradigms can be combined is by using neural networks to help prioritize how symbolic programs organize and search through multiple facts related to a question. For example, if an AI is trying to decide if a given statement is true, a symbolic algorithm needs to consider whether thousands of combinations of facts are relevant.

Such an approach facilitates fast and lifelong learning and paves the way for high-level reasoning and manipulation of objects. The researchers trained this neurosymbolic hybrid on a subset of question-answer pairs from the CLEVR dataset, so that the deep nets learned how to recognize the objects and their properties from the images and how to process the questions properly. Then, they tested it on the remaining part of the dataset, on images and questions it hadn’t seen before. Overall, the hybrid was 98.9 percent accurate — even beating humans, who answered the same questions correctly only about 92.6 percent of the time.

This view then made even more space for all sorts of new algorithms, tricks, and tweaks that have been introduced under various catchy names for the underlying functional blocks (still consisting mostly of various combinations of basic linear algebra operations). Amongst the main advantages of this logic-based approach towards ML have been the transparency to humans, deductive reasoning, inclusion of expert knowledge, and structured generalization from small data. In 2019, Kohli and colleagues at MIT, Harvard and IBM designed a more sophisticated challenge in which the AI has to answer questions based not on images but on videos. The videos feature the types of objects that appeared in the CLEVR dataset, but these objects are moving and even colliding.