This talk was given at the Cologne AI and Machine Learning Meetup on January 23, 2024 (https://www.meetup.com/de-DE/cologne-ai-and-machine-learning-meetup/events/296506410/) by Nico Kreiling: More than the sum of its parts - Building Multi-Agent systems with AutoGen Agents that combine the intelligence of LLMs (Large Language Models) with real code execution are already very powerful. However, they are limited by context and can easily start hallucinating if the prompting is not specific enough. This talk will show how multiple agents can be combined to solve heterogeneous tasks. It will also introduce the AutoGen Framework and discuss the problems associated with using multiple agents.

1. More than the sum of its parts
Building Multi-Agent
systems with AutoGen
Nico Kreiling

2. Nico Kreiling
Nico Kreiling
Data Scientist @ scieneers
Host of techtiefen.de
We gain knowledge from data and create
value. For our customers, society and
ourselves.

3. Most minimal repitition of modern LLMs
Genera&ve Pretrained Transformers (GPT)
à Huge LLMs to predict the next token
Instruc&on Tuning
à Fine Tuning technique to teach language model to
follow user intents
ChatGPT / gpt-3.5-turbo
à GPT3 + instruc@on tuning by OpenAI
Func&on Calling feature in LLMs
à Ability to decide, that it would be beneficial to call
an abstract func@on
Reading-Tip for an intui1ve explana1on of transformers: h9ps://ig.<.com/genera1ve-ai/

4. Agents
• Leverage power of LLMs to implement
custom business logic
• Remember a full conversa@on (statefull)
• Can execute func@ons (i.e. run a cypher
query and return the result
Large-Language-Models
• Trained to do text genera@on
• Does not have any history (stateless)
• Can recommend call a func@on (i.e. to
execute some Cypher-Query)
From Large Language Models to Agents: What are the differences?

5. Autogen enables the seamless creation of flexible multi-
agent conversations
• Open Source project mainly developed by microso6 employees
• Focus on the easy crea9on of llm agent applica9ons in many varia9ons:
- Simple interac9on with one agent
- Explicitley modeled conversa9ons between agents
- Interac9ve Groupchats
• Offers some more nice features such as:
- Caching of repeated requests
- Usage Summary with cost tracking
- Unified model API to use different providers and models
- Hyper Parameter Op9miza9on
- Templa9ng
• Recently Released Autogen Studio UI

6. Autogen offers three types of agents
- UserProxyAgent:
• Represents the user (sends his messages, gets his feedback…)
• Executes func9ons (python interpreter, browsing…)
- AssistantAgent
• An LLM configured with a system message and func9on calling
• The main part to adapt according to own needs
- GroupChatManager
• Used in group conversa9ons to manage message flow
• Can follow simple heuris9cs or use LLM calls to decide to
coordinate between agents

7. How to create a chat with AutoGen?

8. Demo 1

9. Idea: Connect multiple data sources using LLMs
Problem: Big companies have a lot of data in many
different loca@ons and systems.
Combining them into one huge datalake is difficult,
because of different schemas, conven1ons and
technologies used.
Idea: Can we use a chatbot to inui@vely ask ques@ons
regarding all those heterogenous data sources
(knowledge graphs, text-blobs, tables…)?

10. Knowledge Graphs represents a network of real-world
entities and illustrates the relationship between them
Usually stored in Graph Databases
(neo4j, Memgraph, Nebula Graph…),
or RDF stores (Apache Jeno, Amazon
Neptune, Virtuoso…)
Querried by a dedicated language such as
Cypher, SPARQL, Gremlin…

11. The BeTrend Knowledge Graph
Trend
word
Country
Crawl
Related
Topic
Henry Kissinger
Datum: 2023-12-01
Country: Germany
Source: v3wikipediatrends
topic: Nancy Kissinger
type: American philanthropist ‧
Henry Kissinger's wife
eid: Nancy Kissinger
wikiTypes: Agent,Person
wikidataID: Q6962785
wikipedia: http://en.wikipedia.org/
wiki/Nancy_Kissinger
Data Sources:
En3ty

12. Ways to interact with a knowledge graph
Op@on 2: Directly query the knowledge graph
à ChatGPT writes surprisingly good cypher querries when
given a schema and some examples.
Op@on 1: Using predefined querries using APIs:
à Works, but limited flexibility (and boring)
But Problema-c when asked for exact terms that might have different
spellings or formula9ons
Lexical search does not work well though short length
Vector search performs bad as names are mostly unkown to
tokenizer
Solu9on: Using Wikidata search that knows pseudonyms and
takes term popularity into account
Retry mechanism in case of errors, providing the error message with
some hints to the query generator
👱What were the most important trends in Denmark on the first of Octobre 2022?
MATCH (c:CountryCrawl {country: "denmark", day: 1, month: 10, year: 2022})-
[:DAYTREND]-(t:Trendword)
WITH t, t.rank as rank
ORDER BY rank ASC
LIMIT 3
RETURN t.trendword AS popular_trendword

13. BeTrend
Using ChatGPT to interact with the BeTrend-Knowledge-
Graph
• A user asks about trends that relate to Angela
Merkel
• The Coder realizes, that Angela Merkel is an
en9ty, therefore he should look up the wikidata
ID
• Using this wikidata ID, he constructs a cypher
query and passes that one into a func9on call to
run it on the BeTrend Neo4j instance
• If the result is to big for the agents conext size or
empty, corresponding error message will be
passed back to the coder to do another try
• If the cypher query was successful, the agent
reads the JSON response and creates an answer
in natural language
👨💻
👱
Trends related to
Angela Merkel
Angela Merkel =>
Q567
MATCH … => {json}
Angela Merkel related to
the trends…
Cypher-Coder: Your task is to answer questions
using Neo4j. Therefore convert questions to
Cypher queries and run them using provided tools.
Use only the provided relationship types and
properties. Cypher Schema: {schema-info}
User

14. Demo 2

15. Connect multiple data sources using Large-Language-
Models
Title: Scholz weist Erdogan in die Schranken:
Israel-Vorwürfe sind „absurd“!
Source: Bild
Date: 15.11.2023
Vector: [0.234, 0.012, 0.03..
Vector-Datbases:
Used to store ar3cles with full-text
Provides a capable vector-search
op3on to search for similar texts
Has Filtering capabili3es
You are an helpful AI,
that answers user ques@ons
using tools:
🤖
Neo4j: Can run cypher querries.
Helpful for analy@cal ques@ons
regarding en@@es and trends.
Use the following schema when genera@on
cypher code: {schema-info-in-JSON}
Graph-Database:
Used to store different types of data
points (such as trend or en33es) and
their rela3onship
Not intended to store long texts
Could also be used in combina3on
with vector search
Sparql-Querries: [really-complicated-schema….]
Qdrant: A tool to search full text
informa@on from ar@cles
Webbrowsing…

16. Agents
• Leverage power of LLMs to implement
custom business logic
• Remember a full conversa@on (statefull)
• Can execute func@ons (i.e. run a cypher
query and return the result
Divide and Conquer – Combining multiple agents together
Mul&-Agent System
• Configure each agent with a specific system
message and model configura@ons
• Agents use only relevant parts of message
history
• One dedicated model execu@on point

17. How to create a dynamic groupchat with AutoGen?

18. Demo 3

19. 👱
✍
👩💻
👨🔬
👩💼 👨💻
User
I am the coordinator, I
deligate ques;ons
I can help with analy;cal ques;ons
by querying BeTrend as well as full-
text-lookups
I can write ar;cles in
different styles using
given informa;on
I can look up informa;on
in the RTL+ Graph
I construct SPARQL querries for
RTL+ Knowledge graph. Use me
with care!
Future
❓
Poten;ally more sources
to come
Future
What trends every year on the
21. of december
In many years, the beginning of
winter is a trending topic on the 21.
of december

20. 👱
✍
👩💻
👨🔬
👩💼 👨💻
User
I am the coordinator, I
deligate ques;ons
I can help with analy;cal ques;ons
by querying BeTrend as well as full-
text-lookups
I can write ar;cles in
different styles using
given informa;on
I can look up informa;on
in the RTL+ Graph
I construct SPARQL querries for
RTL+ Knowledge graph. Use me
with care!
Future
❓
Poten;ally more sources
to come
Future
Write an arLcle about this in
the style of bild. Use their
previous informaLon.
Darum gibt es zwei Daten für den
Winteranfang…
Wann ist eigentlich Winteranfang?
Top 5 Fakten zum Winteranfang
die du sicher noch nicht kennst…

21. Demo 4

22. Summary: Agents are great, might be even better
Some Take-Aways:
• Agents with code execu9on are really powerfull
• It is easy to get started
• The genera9on of cypher querries works well with a limited schema
Benefits of using Mul-ple Agents
• More fine-grained control on models to use and their configura9on
• Possibility to define dedicated system messages
Downsides of using mul-ple agents:
• AutoGen, as all other LLM frameworks, is s9ll work in progress
• Simply trus9ng the group chat manager to pick the right agent is not
enough
• Models have trouble to evaluate if their answer is sufficient
• Much more requests (therefore poten9ally more expensive)