Agent (lack of) output issues #554

FilterJoe · 2024-09-06T22:36:58Z

FilterJoe
Sep 6, 2024

I've been testing (git clone of) Langroid for the first time in the past couple days, using a Debian 13 VM on a Mac. Poetry worked poorly so I had to install quite a few of the dependencies with pip.

I have it working. The super simple chat demo works fine. Beyond that I've had a few issues, but perhaps it is because I've never bothered to set up a ChatGPT account. I've been using Groq llama-3.1-70b-versatile mostly, but also attempted to use Gemini Pro (which failed miserably, likely due to some kind of end token issue leading to infinite loop) and Cohere Command R+ (did slightly worse than Groq).

The main issue is that when I try to run some of the demos/examples, it often does the actual work (which I can see in the log file) but doesn't actually communicate the result. Is this the intent behind these demos or does this indicate that the LLMs I've chosen aren't working as well as ChatGPT would have? Example:

1-agent-3-tools:

Starting Agent NumberAgent (1) llama-3.1-70b-versatile
TOOL: update
{
"request": "update",
"number": 0
}
Stats: N_MSG=1, TOKENS: in=696, out=21, max=1024, ctx=8000, COST: now=$0.0000,
cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
: update 10
TOOL: update
{
"request": "update",
"number": 10
}
Stats: N_MSG=3, TOKENS: in=720, out=21, max=1024, ctx=8000, COST: now=$0.0000,
cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
: add 7
TOOL: add
{
"request": "add",
"number": 7
}
Stats: N_MSG=5, TOKENS: in=744, out=21, max=1024, ctx=8000, COST: now=$0.0000,
cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
: show
TOOL: show
{
"request": "show"
}
Stats: N_MSG=7, TOKENS: in=766, out=14, max=1024, ctx=8000, COST: now=$0.0000,
cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
:

Sometimes if I try hard enough to convince the LLM to talk to me, it will. But I have noticed that sometimes it does that without even using the tool.

I've had other issues as well but this one is by far the most prevalent. Is there some setting I can play with to steer the agent to be more communicative after getting tool result?

Same thing happened with chat-search which was even more problematic because I would get the JSON showing that it invoked the tool, but no results. And then I would ask it to tell me the results and it would hallucinate a response with made-up links that didn't work (no access to earlier tool result?).

I built a tool of my own for doing one-shot chat-search with llama 3.1 that worked quite well but it took me many days to get it working the way I wanted. It would be great if I could figure out how to use langroid correctly to do the same thing with far fewer lines of code, and with more flexibility than my own little project.

Am I doing something wrong? Are there certain settings I need to play with?

Or does it only really work well with ChatGPT 4 (which I haven't tested yet)?

FilterJoe · 2024-09-07T19:00:29Z

FilterJoe
Sep 7, 2024
Author

I tried chat-search-assistant today and it worked better at communicating to the user than all the other ones I tried. So I can study that one and build from there. But I'm still curious:

Are many of the basic demos intended to demo what's happening internally with little or no communication to the user? Or is user communication intended, but not working due to the prompts being optimized for ChatGPT and working much less well with non-ChatGPT models?

2 replies

pchalasani Sep 7, 2024
Maintainer

@FilterJoe , appreciate you trying out langroid!

with little or no communication to the user?

I'd like to reproduce this -- can you send me the exact cmd you used (i.e. what script and cli args, e.g. -m groq/llama-3.1-70b-versatile) so I make sure I know what you mean by "no communication" ?

Thanks

pchalasani Sep 8, 2024
Maintainer

I tried chat-search-assistant today and it worked better at communicating to the user than all the other ones I tried. So I can study that one and build from there. But I'm still curious:

Are many of the basic demos intended to demo what's happening internally with little or no communication to the user? Or is user communication intended, but not working due to the prompts being optimized for ChatGPT and working much less well with non-ChatGPT models?

I just tried running

python3 examples/basic/chat-search-assistant.py -m groq/llama-3.1-70b-versatile -p metaphor

and it ran fine showing the result to the user at the end:

To answer your question about "lack of communication to user", there are two thing to note:

communication to user is affected by the task setting, e.g. interactive as I said in the other response about 1-agent-3-tools,
indeed a simple multi-agent design suffices for the best models, but falters with weaker models since they often don't use tools correctly. To get good behavior with these weaker models the multi-agent design needs to be modified so each agent is handling a narrow enough task, and there are nudges and corrections for unexpected behavior. For example you can see the variant chat-search-assistant-local.py which is meant to work better with weaker LLMs. The other instructive examples to look at are multi-agent-search-critic and multi-agent-search-critic-no-orch. These are more recent examples that leverage some of the newer orchestration mechanisms. They illustrate two different ways to design a multi-agent system: one leverages langroid's inter-agent orchestration, the other does not. The latter just uses each agent's task as an atomic unit and does not connect them via subtasks, and instead uses normal programming to get the desired result.

pchalasani · 2024-09-08T13:17:18Z

pchalasani
Sep 8, 2024
Maintainer

1-agent-3-tools:

This example is actually working as intended.
The task is set up with interactive=True so it waits for user input at every turn. At the end of the conversation you're showing, where you say show, it correctly uses the show tool, and then waits for user input. As it says in the instruction, the user can hit enter to continue -- if you do that, it will then show you the number.

The alternative way to design the task would be to set it up with interactive = False, which means it won't wait for user input, unless the user the explicitly addressed, but you'd have to modify the agent instructions to explicitly address the user whenever it wants to present something to the user. To see how that can be set up, see this variant I just added: https://github.com/langroid/langroid/blob/main/examples/basic/1-agent-3-tools-address-user.py

3 replies

FilterJoe Sep 8, 2024
Author

Thank you for both of your explanations. I did not interpret the documentation I read correctly.

One thing I read that I found conceptually helpful yesterday was the second blog post:

https://github.com/langroid/langroid/blob/main/docs/blog/posts/langroid-architecture.md

I would suggest linking to that second blog post from the main README. None of the information in that second blog post is obvious to someone who has never worked with an agent framework (or any other type of actor framework) before. I was quite mystified by how everything was working until reading that.

Now that I understand the architecture, combined with your comments in this thread, I should be able to use the Langroid framework to do amazing things.

pchalasani Sep 8, 2024
Maintainer

linking to that second blog post from the main README.

great suggestion, will do!

Now that I understand the architecture, combined with your comments in this thread, I should be able to use the Langroid framework to do amazing things.

Great! If you'd like you can contribute back interesting examples to the repo.

pchalasani Sep 8, 2024
Maintainer

blog post:

here is a blog post rendered in the docs https://langroid.github.io/langroid/blog/2024/08/15/overview-of-langroids-multi-agent-architecture-prelim/
(I assume this is what you read)

FilterJoe · 2024-09-08T23:51:24Z

FilterJoe
Sep 8, 2024
Author

Just tested new variant: 1-agent-3-tools-address-user with Groq llama3.1 70b, but it didn't allow any input at all.

I just got:

WARNING - Task NumberAgent stuck for 5 steps; exiting.

When I set Interactive back to True in then worked, and it did communicate with me (human user) after hitting an enter key. So the following did help make it more communicative:

addressing_prefix=AT

I also modified the system prompt slightly so that it wouldn't call me "|@|User, but would instead call me by my real name. Worked first try:

system_message=f"""
When the user's request matches one of your available tools, use it, 
otherwise respond directly to the user.
NOTE: Whenever you want to address the user directly, you MUST
use "{AT}User", followed by your message. Try once to learn the human user's actual name and once you know the
human user's name, use it in place of User.
"""

1 reply

pchalasani Sep 9, 2024
Maintainer

That is odd. I just tried it again with this model and it runs fine...

pchalasani · 2024-09-09T00:43:15Z

pchalasani
Sep 9, 2024
Maintainer

@FilterJoe another way to have the system wait for the user, is to use the new ForwardTool, e.g. in 2-agent-tools.py and elsewhere.
This avoids the need for instructing the LLM to explicitly address the user. This would only work in scenarios where it's clear that any non-tool msg from the LLM is intended for the user.

0 replies

FilterJoe · 2024-09-09T02:09:11Z

FilterJoe
Sep 9, 2024
Author

I must have changed some setting somewhere that's causing this and forgot what my change was. I'll stop bothering you with questions for a few days until I get a better handle on things as I continue to read and understand the many different parameters and how the code is structured.

The code quality is really great. Part of what I why I picked langroid for my first agent framework is to study high quality code. I hope to adopt (what looks to me like) high quality coding style and design patterns into my own coding projects.

Maybe I can come up with a good example or two to add to the project.

Thank you for all your help!

6 replies

FilterJoe Sep 10, 2024
Author

I did a completely fresh install of a new VM, new dev(code) version of langroid to try to see if I'm still having the same issues. I did have the same issues: when only running one task with one agent, the LLM is reluctant to give output to the human user (me). And again with 1-agent-3-tools-address-user (using Groq llama3.1 70b), I get:

WARNING - Task NumberAgent stuck for 5 steps; exiting.

And I never had a chance to give input.

I'm wondering if you have something installed on your system that is a dependency that I'm missing. Details of the fresh install:

Mac Mini M2 Pro 16GB RAM
Debian 12 VM with 8GB RAM
PyCharm editor (but same result running in the PyCharm console or using the terminal)
groq/llama-3.1-70b-versatile

My .env looks like this with secrets removed:

OPENAI_CHAT_MODEL=groq/llama-3.1-70b-versatile
GROQ_API_KEY=mykey
COHERE_API_KEY=mykey
WOLFRAM_API_KEY=mykey
BRAVE_API_KEY=mykey
GEMINI_API_KEY=mykey
SERPER_API_KEY=mykey
GOOGLE_API_KEY=mykey
GOOGLE_CSE_ID=mykey
REDIS_PASSWORD=mypwd
REDIS_HOST=redis…..redis-cloud.com
REDIS_PORT=portnumber

Maybe something listed above is good enough to suggest what I may be missing compared to your environment. If not, I recorded very detailed instructions for creating the VM which I could share with you if you find that helpful or useful.

Aside: I solved the poetry issue and got it to work. Don't know if you want to add this to the docs, but need following dependencies installed:

sudo apt-get install pkg-config python3-dev default-libmysqlclient-dev build-essential

pchalasani Sep 10, 2024
Maintainer

@FilterJoe can you show:

the output of pip show langroid, or what langroid version you're using (latest is 0.15.0)
screenshot/text of output when you run the script.

FilterJoe Sep 10, 2024
Author

I'm surprised that my langroid version is 0.14 as I did a git pull less than 24 hours ago, using the following URL:

git clone https://github.com/langroid/langroid.git

Here is output of pip show langroid (with your email x'ed out):

(langroid-py3.12) jg@d1:/PycharmProjects/langroid$ pip show langroid
Name: langroid
Version: 0.14.0
Summary: Harness LLMs with Multi-Agent Programming
Home-page:
Author: Prasad Chalasani
Author-email: xxxxx
License: MIT
Location: /home/jg/PycharmProjects/langroid/.venv/lib/python3.12/site-packages
Editable project location: /home/jg/PycharmProjects/langroid
Requires: aiohttp, async-generator, bs4, colorlog, docstring-parser, duckduckgo-search, faker, fakeredis, fire, google-api-python-client, google-generativeai, groq, grpcio, halo, jinja2, json-repair, lxml, nest-asyncio, nltk, onnxruntime, openai, pandas, prettytable, pydantic, pygithub, pygments, pyparsing, python-dotenv, python-magic, pyyaml, qdrant-client, rank-bm25, redis, requests, requests-oauthlib, rich, thefuzz, tiktoken, trafilatura, typer, wget
Required-by:
(langroid-py3.12) jg@d1:~/PycharmProjects/langroid$

FilterJoe Sep 10, 2024
Author

(langroid-py3.12) jg@d1:~/PycharmProjects/langroid$ PYTHONPATH=$(pwd) python3 examples/basic/1-agent-3-tools-address-user.py

Starting Agent NumberAgent (1) llama-3.1-70b-versatile
(cached)I'm ready to assist you. What would you like to do?
Stats: N_MSG=1, TOKENS: in=0, out=0, max=1024, ctx=8000, COST: now=$0.0000, cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
2024-09-10 06:36:24 - WARNING - Task NumberAgent stuck for 5 steps; exiting.
Bye, hope this was useful!
<<< Finished Agent NumberAgent (2)

pchalasani Sep 10, 2024
Maintainer

In this example the LLM was supposed to explicitly address the user using |@|User but forgot to do it, which caused the stuck-for-5-steps. In the updated example, any LLM msg with no tool is forwarded to the user, using the handle_message_fallback method and a ForwardTool.

FilterJoe · 2024-09-10T13:39:52Z

FilterJoe
Sep 10, 2024
Author

(langroid-py3.12) jg@d1:~/PycharmProjects/langroid$ PYTHONPATH=$(pwd) python3 examples/basic/1-agent-3-tools.py

Starting Agent NumberAgent (1) llama-3.1-70b-versatile
(cached)TOOL: show
{
"request": "show"
}
Stats: N_MSG=1, TOKENS: in=0, out=0, max=1024, ctx=8000, COST: now=$0.0000, cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
: update 50
TOOL: update
{
"request": "update",
"number": 50
}
Stats: N_MSG=3, TOKENS: in=700, out=21, max=1024, ctx=8000, COST: now=$0.0000, cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
: add 10
TOOL: add
{
"request": "add",
"number": 10
}
Stats: N_MSG=5, TOKENS: in=724, out=21, max=1024, ctx=8000, COST: now=$0.0000, cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
: show
TOOL: show
{
"request": "show"
}
Stats: N_MSG=7, TOKENS: in=746, out=14, max=1024, ctx=8000, COST: now=$0.0000, cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
:
TOOL: show
{
"request": "show"
}
Stats: N_MSG=9, TOKENS: in=770, out=14, max=1024, ctx=8000, COST: now=$0.0000, cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
: update 100
TOOL: update
{
"request": "update",
"number": 100
}
Stats: N_MSG=11, TOKENS: in=787, out=21, max=1024, ctx=8000, COST: now=$0.0000, cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
:
TOOL: show
{
"request": "show"
}
Stats: N_MSG=13, TOKENS: in=817, out=14, max=1024, ctx=8000, COST: now=$0.0000, cumul=$0.0000, tot=$0.0000 (llama-3.1-70b-versatile)
Human (respond or q, x to exit current level, or hit enter to continue)
: q
Bye, hope this was useful!
<<< Finished Agent NumberAgent (14)

1 reply

pchalasani Sep 10, 2024
Maintainer

I changed this script to use interactive=False, meaning it waits for user input only if explicitly addressed. Of course there's no addressing mechanism or instructions, so the LLM would never explicitly address the user. Instead, the handle_message_fallback detects when there's a non-tool LLM msg and forwards to the user using the ForwardTool. Also the repeated tool-generation is a weak-LLM deviation, so to mitigate this I adjusted the tool-handler responses to nudge the LLM to just ask the user what they want.

FilterJoe · 2024-09-10T13:41:32Z

FilterJoe
Sep 10, 2024
Author

In that last run you can see that sometimes after the : is empty when I just hit enter, and again it just showed JSON of the request.

0 replies

FilterJoe · 2024-09-10T15:05:22Z

FilterJoe
Sep 10, 2024
Author

I think the langroid 0.14 is coming from conda which is installed on that VM. Not sure if that is relevant given the python source code being present.

The first VM I used to test langroid did not have conda, and therefore did not have langroid at all when I did pip show langroid. It was just running off the python code I pulled from git.

0 replies

pchalasani · 2024-09-10T16:19:50Z

pchalasani
Sep 10, 2024
Maintainer

@FilterJoe I updated the examples so you can pass the model as a cmd line arg using -m <model>.

In my previous examples I showed you, I was passing these args but they were ignored since the code wasn't set up to use them.
I made other changes in the two examples as well. They are both set up as non-interactive, and there is a fallback handler that sends any non-tool msg to the user.

As for langroid, you should be able to update to the latest (I never use conda, so can't advise there).

python3 examples/basic/1-agent-3-tools.py -m groq/llama-3.1-70b-versatile

gives this --

and

python3 examples/basic/1-agent-3-tools-address-user.py -m groq/llama-3.1-70b-versatile

gives this --

0 replies

FilterJoe · 2024-09-10T17:07:23Z

FilterJoe
Sep 10, 2024
Author

It's fixed after git pull, poetry install.

My system is now running just like yours.

I'll let you know if I run into any other unexpected behaviors.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Agent (lack of) output issues #554

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 10 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Agent (lack of) output issues #554

FilterJoe Sep 6, 2024

Replies: 10 comments · 13 replies

FilterJoe Sep 7, 2024 Author

pchalasani Sep 7, 2024 Maintainer

pchalasani Sep 8, 2024 Maintainer

pchalasani Sep 8, 2024 Maintainer

FilterJoe Sep 8, 2024 Author

pchalasani Sep 8, 2024 Maintainer

pchalasani Sep 8, 2024 Maintainer

FilterJoe Sep 8, 2024 Author

pchalasani Sep 9, 2024 Maintainer

pchalasani Sep 9, 2024 Maintainer

FilterJoe Sep 9, 2024 Author

FilterJoe Sep 10, 2024 Author

pchalasani Sep 10, 2024 Maintainer

FilterJoe Sep 10, 2024 Author

FilterJoe Sep 10, 2024 Author

pchalasani Sep 10, 2024 Maintainer

FilterJoe Sep 10, 2024 Author

pchalasani Sep 10, 2024 Maintainer

FilterJoe Sep 10, 2024 Author

FilterJoe Sep 10, 2024 Author

pchalasani Sep 10, 2024 Maintainer

FilterJoe Sep 10, 2024 Author

FilterJoe
Sep 6, 2024

Replies: 10 comments 13 replies

FilterJoe
Sep 7, 2024
Author

pchalasani Sep 7, 2024
Maintainer

pchalasani Sep 8, 2024
Maintainer

pchalasani
Sep 8, 2024
Maintainer

FilterJoe Sep 8, 2024
Author

pchalasani Sep 8, 2024
Maintainer

pchalasani Sep 8, 2024
Maintainer

FilterJoe
Sep 8, 2024
Author

pchalasani Sep 9, 2024
Maintainer

pchalasani
Sep 9, 2024
Maintainer

FilterJoe
Sep 9, 2024
Author

FilterJoe Sep 10, 2024
Author

pchalasani Sep 10, 2024
Maintainer

FilterJoe Sep 10, 2024
Author

FilterJoe Sep 10, 2024
Author

pchalasani Sep 10, 2024
Maintainer

FilterJoe
Sep 10, 2024
Author

pchalasani Sep 10, 2024
Maintainer

FilterJoe
Sep 10, 2024
Author

FilterJoe
Sep 10, 2024
Author

pchalasani
Sep 10, 2024
Maintainer

FilterJoe
Sep 10, 2024
Author