ISWC-2024-Tutorial-Neurosymbolic-Customized-and-Compact-CoPilots

Neurosymbolic Customized and Compact CoPilots

ℹ️ Authors: Kaushik Roy, Megha Chakraborty, Yuxin Zi, Manas Gaur, and Amit Sheth

📰 Paper: Neurosymbolic Customized and Compact CoPilots

Tutorial Details

📅 Date and time

Nov 12th 2024, 11am-12:40pm

✈️ Venue

International Semantic Web Conference (ISWC), 2024

📎 Slides

📹 Video

💻 Code

Abstract

Large Language Models (LLMs) are credible with open-domain interactions such as question answering, summarization, and explanation generation [1]. LLM reasoning is based on parametrized knowledge, and as a consequence, the models often produce absurdities and inconsistencies in outputs (eg, hallucinations and confirmation biases)[2]. In essence, they are fundamentally hard to control to prevent off-the-rails behaviors, are hard to fine-tune, customize for tailored needs, prompt effectively (due to the “tug-of-war” between external and parametric memory), and extremely resource-hungry due to the enormous size of their extensive parametric configurations [3, 4]. Thus, significant challenges arise when these models are required to perform in critical applications in domains such as healthcare and finance, that need better guarantees and in turn, need to support grounding, alignment, and instructibility. AI models for such critical applications should be customizable or tailored as appropriate for supporting user assistance in various tasks, compact to perform in real-world resource-constraint settings, and capable of controlled, robust, reliable, interpretable, and grounded reasoning (grounded in rules, guidelines, and protocols)[5]. This special session explores the development of compact, custom neurosymbolic AI models and their use through human-in-the-loop co-pilots for use in critical applications [6].

References

1. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
2. Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922, 2023.
3. Kevin Wu, Eric Wu, and James Zou. How faithful are rag models? quantifying the tug-of-war between rag and llms’ internal prior. arXiv preprint arXiv:2404.10198, 2024.
4. Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Qiuxia Li, and Jun Zhao. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models. arXiv preprint arXiv:2402.14409, 2024.
5. Amit Sheth, Manas Gaur, Kaushik Roy, Revathy Venkataraman, and Vedant Khandelwal. Process knowledge-infused ai: Toward user-level explainability, interpretability, and safety. IEEE Internet Computing, 2022.
6. Amit Sheth, Kaushik Roy, and Manas Gaur. Neurosymbolic artificial intelligence (why, what, and how). IEEE Intelligent Systems, 38(3):56–62, 2023.

🤖 Copilot Examples

🥣 Nourish Co-pilot: Custom, Compact, and NeuroSymbolic Diet AI Model

Can I eat this food or not? Does it have ingredients I’m allergic to? How can I make this more nutritious or vegan or keto-free? Does this recipe conform to diabetes guidelines? These are simple yet powerful questions that we ask about our food. To answer these questions and more, we introduce:

We introduce 𝙉𝙤𝙪𝙧𝙞𝙘𝙝: 𝘼 𝘾𝙪𝙨𝙩𝙤𝙢, 𝘾𝙤𝙢𝙥𝙖𝙘𝙩 𝙖𝙣𝙙 𝙉𝙚𝙪𝙧𝙤𝙨𝙮𝙢𝙗𝙤𝙡𝙞𝙘 𝘿𝙞𝙚𝙩 𝘼𝙄 𝙢𝙤𝙙𝙚𝙡 - That analyses the suitability of a given recipe by analyzing ingredients and cooking actions in multiple contexts. The system also provides explanations in the form of reasoning to the users. Given a recipe is not suitable, the system aims to provide alternative recipes or ingredient substitutions.

Visit here for the demo: Demo link

🔧 𝘾𝙪𝙨𝙩𝙤𝙢: Tailored to analyze and reason over the suitability of a recipe for diabetes and provide alternative recipes or ingredient and cooking action substitutions.

⚙️ 𝘾𝙤𝙢𝙥𝙖𝙘𝙩: Lightweight and cost-effective model, optimized for real-time deployment on consumer-grade hardware

🧠 𝙉𝙚𝙪𝙧𝙤𝙨𝙮𝙢𝙗𝙤𝙡𝙞𝙘: An explainable food recommendation framework that adapts semantic, perceptual, and cognitive framework to ground data with semantic knowledge (semantics), mapping grounded data to disease context (perceptual) and provides reasoning for the recommendation(cognitive) to the users along with the source of knowledge utilized for recommendations.

❤️ MTSS Co-pilot: Custom, Compact and Neurosymbolic Behavioral Health Assistance Management

The Multi-Tiered System of Support (MTSS) framework is designed to assist students with behavioral strategies by drawing on the support of a network that includes parents, counselors, and school administrators. 🚀 However, the intricate nature of the MTSS structure can overwhelm the support network when addressing a student’s case due to the numerous options available. 🚀 We introduce MTSS-CoPilot, a digital assistant designed to provide essential decision support to the members of a student’s support network. This assistant employs an innovative AI approach that ensures its outputs are tailored to the specific needs of the support network, align with expert knowledge, and remain transparent and explainable.

Visit here for the demo: Demo link

🔧 𝘾𝙪𝙨𝙩𝙤𝙢: Tailored to solve specific industry challenges (here focused on student-behavioral health support leveraging various members in the student’s support network), providing focused and practical solutions.

⚙️ 𝘾𝙤𝙢𝙥𝙖𝙘𝙩: Lightweight and cost-effective, optimized for real-time deployment on consumer-grade hardware.

🧠 𝙉𝙚𝙪𝙧𝙤𝙨𝙮𝙢𝙗𝙤𝙡𝙞𝙘: Utilizes advanced reasoning capabilities over carefully curated assets, including specialized data, domain knowledge, and human expertise. This ensures the system delivers reliable and safe outputs by adhering to these curated resources.