Integrating OpenAI with React Native: A Fun Experiment

I recently developed a simple application using OpenAI/ChatGPT and was amazed by how enjoyable and straightforward the process was.

Watch the app preview here

While the app is admittedly more of a novelty than a serious tool, I was particularly impressed by the ability to upload images and ask specific questions about them. This was my first experience with AI integration, and though it might seem obvious to developers already working with AI models, I found it fascinating.

Let's examine the core communication function to better understand how the app works:

const sendMessage = async (image_url: string, language: AppLanguage) => {
  try {
    const openai = new OpenAI({
      apiKey: OPENAI_API_KEY, // This is the default and can be omitted
    });

    const { data: completion } = await openai.chat.completions
      .create({
        model: "gpt-4-turbo",
        stream: false,
        messages: [
          {
            role: "system",
            content: `
You are palm reader, explain by reading palm the user background, life, future, fate. Please write at least 10 setences about each topic:
- user line of life
- user line of heart
- user line of mind
- user line of fate
- user extra information that we need to mention

Only provide a RFC8259 compliant JSON response:
[
  {
    "line_life": "user line of life",
    "line_heart": "user line of heart",
    "line_mind": "user line of mind",
    "line_fate": "user line of fate",
    "line_extra": "user extra information"
  }
]`,
          },
          {
            role: "user",
            content: [
              {
                type: "text",
                text: `
- user line of life, 10+ sentences
- user line of heart, 10+ sentences
- user line of mind, 10+ sentences
- user line of fate, 10+ sentences
- user extra information that we need to mention, 10+ sentences
                `,
              },
              {
                type: "text",
                text: `Use language ${language} for answers.`,
              },
              {
                type: "text",
                text: 'Describe my palm but in format { "line_heart": "text...", "line_life": "text...", "line_mind": "text...", "line_fate": "text...", "line_extra": "text..." }',
              },
              {
                type: "image_url",
                image_url: {
                  url: `data:image/jpeg;base64,${image_url}`,
                },
              },
            ],
          },
        ],
      })
      .withResponse();

    const replies = completion.choices
      .flatMap((choice) => {
        try {
          if (choice.message.content === null || choice.message.content === undefined) {
            return [];
          } else {
            const messages = JSON.parse(choice.message.content);

            return messages;
          }
        } catch (error) {
          Sentry.Native.captureException(error);

          return [];
        }
      })
      .reduce((acc, cur) => ({ ...acc, ...cur }), {});

    return replies;
  } catch (error) {
    Sentry.Native.captureException(error);
  }
};

As you can see, I used specific prompts to format the responses in JSON and passed an image for analysis. I also localized the app by supporting English, Spanish, German, French, and Russian languages. Simply instructing ChatGPT to respond in a specific language worked perfectly out of the box—it's like mathematical magic!

I plan to publish the app, but there's a significant challenge: using gpt-4-turbo is quite expensive. The app will likely remain available for a while, but I may need to disable it eventually due to the costs involved.