This project is a web-based application that uses a Gemma model to translate natural language commands into turtle graphics (Logo). You can ask the agent to draw shapes, patterns, and more, and watch the turtle draw it on the canvas. You can also manually write and run Logo code.
- Natural Language to Logo: Translate commands like "draw a blue square" into Logo code.
- Voice Commands: Record audio directly in the browser to describe what you want the turtle to draw.
- AI Agent: Uses a Gemma model with tool-calling capabilities to generate graphics commands.
- Interactive Web UI: Built with Gradio for an easy-to-use interface.
- Live Canvas: See the turtle draw on the canvas in real-time.
- Manual Mode: Write and run your own Logo code.
- Model Selection: Choose from different Gemma models.
The application follows these steps:
- User Input: The user enters a natural language command (e.g., "draw a red circle") or records a voice request via the microphone in the Gradio interface.
- Gemma Agent: The
GemmaAgenttakes the input and, using a pre-defined set of tools, generates a series of turtle graphics commands. - Turtle Engine: The
HeadlessTurtleengine executes these commands, drawing the output on a PIL image. - UI Update: The Gradio UI is updated to show the generated Logo code, the final image, and the agent's logs.
ββββββββββββββββββ βββββββββββββββ βββββββββββββββββββ βββββββββββββββββ
β User Interface βββββ>β Gemma Agent βββββ>β Turtle Engine βββββ>β Output Image β
β (Gradio) β<βββββ (LLM) β<βββββ (PIL) β<βββββ (Canvas) β
ββββββββββββββββββ βββββββββββββββ βββββββββββββββββββ βββββββββββββββββ
-
Clone the repository:
git clone <repository-url> cd <repository-directory>
-
Create a virtual environment (recommended):
python3 -m venv venv source venv/bin/activate -
Install the dependencies:
pip install gradio torch transformers bitsandbytes Pillow accelerate scipy soundfile librosa
To start the application, run the app.py file:
python app.pyThis will start a local Gradio server. Open the URL provided in your terminal to access the web interface.
app.py: The main application file that runs the Gradio web UI.gemma_agent.py: Manages the Gemma model, including loading the model, processing text and audio prompts, and generating tool calls.turtle_engine.py: A "headless" turtle graphics engine that draws on a PIL image. It keeps track of the turtle's state (position, heading, etc.) and command history.logo_interpreter.py: A simple interpreter for standard Logo commands (e.g.,FD,RT,REPEAT). It translates these commands into actions for theHeadlessTurtle.config.py: Contains configuration for the application, such as the default Gemma model ID and the tool definitions for the AI agent.test.py: A command-line script for testing theGemmaAgentwithout the Gradio UI.
