Lost in Translation: Getting Llama 2 to Speak Python

I'm going to write about my AI travails using PALChain and Llama 2 to get my thoughts straight and, who knows, perhaps help you in the process. I've had worse ideas.

In my previous post, I mentioned that PALChain doesn't work (for me) with Llama 2. That's frustrating, because it does work with GPT. And I want to rely on local Large Language Models. Because reasons.

= PALChain =

Let me illustrate what I mean. For those unfamiliar, Program Aided Language models (which PALChain helps you work with) offload maths to Python or another development environment, leaving the LLM to what it does best: talk. So if you ask it for the solution to an equation such as 2 + 3, instead of risking the wrong answer, GPT will devise a Python function:

```

def solution():

x = 2

y = 3

result = x + y

return result

```

and LangChain will get Python to run that function. Ideally, '5' will be the LLM's reply.

When I run this past OpenAI, it works flawlessly: I prompt it "If my age is half of my dad's age and he is going to be 60 next year, what is my current age?", and it returns a terse and correct '29.5'. When I do the same with Llama 2 on Ollama, I get a long and cryptic error message, ending in "ValueError: Generated code is not valid python code:", plus what appears to be valid Python code.

My initial thought was that I'd messed up my code. I spent long hours combing through my code, comparing it to other code that did work. To no avail: the error message remained.

[TO DO: learn how to read error messages]

= Open Interpreter =

As I went through this several times, it hit me that I actually encountered this very problem when I tried to set up Open Interpreter, a local alternative to ChatGPT's Advanced Code Analysis. Open Interpreter leverages an LLM's language abilities to generate code for you, as if you had a developer buddy sitting next to you. Empowering, when it works.

You see, Open Interpreter gives you the option to pass your prompts to GPT using the OpenAI API (for pay), which is quick and effective. You can also opt for Llama 2 installed locally, which in my case resulted in an error message that I don't remember because I tried it a month ago. At the time, I thought nothing of it and blamed myself. Now that I couldn't get PALChain to run either, it hit me that I should ask an expert.

= Prompts =

"There might be some differences in how the local LLM handles text input or generates text output compared to the OpenAI API," ChatGPT Advanced Data Analysis replied. The way that I interpret this is, the prompt that works with ChatGPT does not work with Llama 2. I tried various prompt alternatives:

- 'please generate a Python function'

- 'only return the Python function function'

- ending the prompt with '=== def solution():'

and other, some of which I came up with and some suggested by ChatGPT, but the output of Llama 2 kept including more than just the function definition.

[TO DO: learn how to read LLM documentation]

= Make it =

I'm stuck, I'll admit. Writing this has cleared things up in my mind somewhat, but I don't have a solution. I have come to the conclusion that Llama 2 does not give the expected reply. This could be due to a difference in how it interprets prompts, possibly because it has been trained differently. I'm happy to have learned it's most likely not a problem with my code; when I get an error message, I switch to panic mode and assume errors on my side. ChatGPT helped me understand that the model may be the source of the issue.

Perhaps this post has given you some ideas on how to tackle similar issues, hopefully with better results. If you've faced this before and found a solution, I'd love to hear your thoughts. In the meantime, hang on in there and you just might make it!

5 comments