OpenAI Code

Visual Studio Code

As we use Visual Studio Code to compile and upload code, we need to install the software before proceeding.

Windows

First, download Visual Studio Code by visiting https://code.visualstudio.com/Download. Choose the appropriate version for your operating system, then download and install it

../../../_images/Chapter02_001.png

Double-click the downloaded .exe file to run it.

Check the box for “I accept the agreement.”

Then click Next.

../../../_images/Chapter02_011.png

The installation location can be left as the default or changed to a desired path. After that, proceed by clicking Next repeatedly.

../../../_images/Chapter02_021.png

On this screen, verify that “Add to PATH” is selected. If unchecked, enable it. Proceed by clicking Next repeatedly to finish the installation.

../../../_images/Chapter02_031.png

The installation is now complete, as shown in the image below.

../../../_images/Chapter02_04.png

Mac

Typically, MacOS comes with Visual Studio Code pre-installed. If your computer does not have it, please install it first.

Visit https://code.visualstudio.com and click “Download for macOS” .

../../../_images/Chapter02_051.png

Double click to run the program.

../../../_images/Chapter02_061.png

Linux

If your computer does not have Visual Studio Code, please install it first.

Visit https://code.visualstudio.com and click “.deb” .

../../../_images/Chapter02_071.png

Open the downloaded”code_xxx.deb” file.

../../../_images/Chapter02_081.png

Click “Install” to install Visual Studio Code.

../../../_images/Chapter02_09.png

Wait for the installation to complete. Once finished, it should look like the image below.

../../../_images/Chapter02_10.png

Click Show Apps and you can see the Visual Studio Code is in the system.

../../../_images/Chapter02_11.png

ESP-IDF V5.4.1

Visual Studio Code is a versatile code editor. To program with the ESP-IDF SDK, we need to install the ESP-IDF extension for it.

Open Visual Studio Code.

Click on the menu bar: File -> Preferences -> Extensions.

../../../_images/Chapter02_12.png

Mac OS: Click “Code”->”Preferences”->”Extensions” on the menu bar.

../../../_images/Chapter02_13.png

Search for “ESP-IDF” in the extension bar, select the correct result from the list, then click the Install button to proceed.

../../../_images/Chapter02_14.png

The ESP-IDF extension icon will now appear in the left sidebar - click it to continue.

../../../_images/Chapter02_15.png

Scroll down with your mouse, locate and click on the “Advanced” option.

../../../_images/Chapter02_16.png

Click the first option: “Configure ESP-IDF Extension”.

../../../_images/Chapter02_17.png

Select “EXPRESS” on the right.

../../../_images/Chapter02_18.png

Note

If you’re using macOS or Ubuntu, please complete the necessary preparations as prompted before proceeding with installation.

../../../_images/Chapter02_19.png

  1. Check the box for “Show all ESP-IDF tags”

  2. Select “v5.4.1 (release version)” from the dropdown

  3. Choose your desired installation path for the ESP-IDF environment

  4. Click “Install” to begin the setup

The installation path varies among computer systems, please remember it.

../../../_images/Chapter02_20.png

The process will complete automatically.

If it failed, locate your chosen ESP-IDF directory, remove the failed installation folder and install it again.

../../../_images/Chapter02_21.png

This step may take a while, so please ensure you have a stable and fast internet connection.

If the installation continues to fail, check the relevant link for your operating system below:

Window: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/windows-setup.html

Mac & Linux: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/linux-macos-setup.html

The complete installation is as shown below.

../../../_images/Chapter02_22.png

For more about ESP-IDF, please refer to:

https://docs.espressif.com/projects/vscode-esp-idf-extension/en/latest/installation.html

OpenAI Code

This project is derived from the open-source repository: https://github.com/openai/openai-realtime-embedded, licensed under MIT License. We have only adapted it for third-party learning and AI functionality trials, without any commercial promotion or application. This tutorial is intended solely for enthusiasts to supplement their learning.

Code Downloading

Windows

Use the shortcut “Win+R”, enter “CMD” in the pop-up window, and open the CMD interface.

In the Terminal, install the code with git command.

git clone --recurse-submodules https://github.com/Freenove/openai-realtime-embedded
../../../_images/Chapter02_23.png

The installation is completed as shown below.

../../../_images/Chapter02_24.png

If you do not have the git tool on your computer, please download and install it by visiting https://git-scm.com/downloads

MAC & Linux

Open the Terminal, run the following command, and wait for the installation to finish.

git clone --recurse-submodules https://github.com/Freenove/openai-realtime-embedded
../../../_images/Chapter02_25.png

The installation completes as shown below.

../../../_images/Chapter02_26.png

Configuring Code Environment

On Visual Studio Code, click “File” -> “Open Folder…”

../../../_images/Chapter02_27.png

Select the openai-realtime-embedded older. Here, the interface of the Windows system is taken as an example. The operation on the mac and Linux system is similar.

../../../_images/Chapter02_28.png

Check the box “Trust the authors of all files in the parent folder “Downloads” and select “Yes, I trust the authors”.

../../../_images/Chapter02_29.png

Please note: A pop-up notification ‘Generate comple_commands.json’ will appear in the lower-right corner.

Please disregard it. DO NOT click it.

../../../_images/Chapter02_75.png

Connect the ESP32S3 WROOM board to your computer with the USB cable (do not use the wrong connector).

../../../_images/Chapter02_34.png

Click on ‘COMx’ in the bottom-left corner to display all available COM ports on your computer. Locate and select the entry labeled ‘ESP32-S3’.

../../../_images/Chapter02_76.png

Click the ‘ESP32’ button in the bottom-left corner to display all available ESP32 models, then select ‘ESP32S3’ from the list.”

../../../_images/Chapter02_77.png

From the new selection menu, choose ‘ESP32-S3 Chip (via ESP-PROG) - ESP32-S3 debugging via ESP-PROG Board…’

../../../_images/Chapter02_78.png

Wait until it shows “Target ESP32S3 Set Successfully” at the bottom right.

../../../_images/Chapter02_79.png

As of the time of writing, the Freenove Media Kit for ESP32-S3 comes in two different models with varying screen sizes. However, this tutorial applies to both versions. Please select the corresponding macro definition based on your device’s screen size (only one should be enabled, and the other must be commented out).

../../../_images/Chapter02_74.png

Registering Open API Keys

If you don’t have an OpenAI API key yet, please visit https://platform.openai.com/docs/overview to sign up and purchase one.

You’ll need to create an account. Click “Sign up” in the top-right corner.

../../../_images/Chapter02_43.png

Create the account in any way you prefer.

../../../_images/Chapter02_44.png

Here we take Email address as an example. Input the email address and password, and click Continue.

../../../_images/Chapter02_45.png

OpenAI Platform will send a verification code to your email address. Input the code to the bar, click Continue.

../../../_images/Chapter02_46.png

Complete your personal information.

../../../_images/Chapter02_47.png

You can either fill the Organization Name or not, then click Create organization.

../../../_images/Chapter02_48.png

You can click “I’ll invite my team later”.

../../../_images/Chapter02_49.png

Name it whatever you like, and then click “Generate API Key”.

../../../_images/Chapter02_50.png

Click Copy and paste your “My test key” somewhere to save it. If you click “Continue”, you will no longer be able to view the key again!

We recommend creating a separate document to store your key securely and prevent loss.

../../../_images/Chapter02_51.png

OpenAI operates on a paid service model. You need to purchase for credits to ensure uninterrupted access to its features.

../../../_images/Chapter02_52.png

After completion, the interface will appear as shown below. Click the settings icon in the upper right corner.

../../../_images/Chapter02_53.png

You can click “API Keys” to manage your OpenAI keys.

../../../_images/Chapter02_54.png

Click “Billing” to check your available balance, or click “Add Payment Details” to top up funds.

../../../_images/Chapter02_55.png

Now you have your own working OpenAI API keys. Feel free to explore more features on the website.

Code Compilation and Uploading

After ensuring everything is properly configured, let’s start compiling the code.

Click “Full Clean” below to clear all previous compilation data.

../../../_images/Chapter02_57.png

Click “ESP-IDF: Build, Flash and Monitor” below. It will compile the code, upload it to the ESP32-S3, and open the serial monitor.

../../../_images/Chapter02_58.png

The first compilation may take longer—please ensure a stable internet connection and wait patiently.

If you encounter the following error during compilation, it’s typically caused by issues with the ESP-IDF toolchain. This often occurs due to network problems that prevented some toolchain components from installing properly. Please reinstall ESP-IDF. You can refer to ESP-IDF V5.4.1.

../../../_images/Chapter02_59.png

When the code compilation is complete and you upload it for the first time, you will see the prompt below. Please select “UART”.

../../../_images/Chapter02_60.png

After the upload completes, the serial monitor will automatically open and establish an internet connection to access OpenAI. You can now start chatting via the ESP32-S3.

../../../_images/Chapter02_61.png

Computer End

Enable Wi-Fi, locate “OpenAI” in the Wi-Fi settings, and click “Connect”.

../../../_images/Chapter02_62.png

Open the browser, enter 192.168.4.1 in the address bar, and press Enter to access the network configuration page.

../../../_images/Chapter02_63.png

Enter the Wi-Fi name and password, as well as the API key of OpenAI, and then click Submit.

../../../_images/Chapter02_64.png

The following interface indicates successful configuration.

../../../_images/Chapter02_65.png

When you see the “How can I help” prompt, you can start communicate with openAI.

../../../_images/Chapter02_66.png

The chat logs will be displayed simultaneously in VS Code’s Serial Monitor.

../../../_images/Chapter02_67.png

Mobile End

Enable Wi-Fi, locate “OpenAI” in the Wi-Fi settings to connect.

../../../_images/Chapter02_68.png

Open the browser on your phone, input 192.168.4.1 in the address bar to enter the network settings interface.

../../../_images/Chapter02_73.png

Enter the Wi-Fi name and password, as well as the API key of OpenAI, and then click Submit.

../../../_images/Chapter02_69.png

The following interface indicates successful configuration.

../../../_images/Chapter02_70.png

When you see the “How can I help” prompt, you can start communicate with openAI.

../../../_images/Chapter02_71.png

The chat logs will be displayed simultaneously in VS Code’s Serial Monitor.

../../../_images/Chapter02_72.png