Skip to content
This repository was archived by the owner on Dec 9, 2025. It is now read-only.
This repository was archived by the owner on Dec 9, 2025. It is now read-only.

Question: How to perform OCR on Camera's live frame data? #16

Closed

Description

@Kommandat

Really more of a question (or a feature request), but is it possible to perform OCR on the live frame data from the Camera?

In my app I was trying to get around this by repeatedly taking pictures at regular intervals and scanning them to show red bounding boxes for the text (video below). This would've worked fine, except iOS makes a loud shutter noise when you take a picture (and a shutter noise every 500ms is really really annoying).

It looks like expo-camera (which is what I'm using) doesn't access the live frame data but there are some other options out there that do, but I don't know how to get them to work with MLKit:

Thanks! Appreciate this package a ton and understand what I'm asking here might be difficult to do. If so, I'm also curious to learn more and help out.

RPReplay_Final1632580413.mov

Activity

  1. agoldis commented on Sep 25, 2021

    @agoldis
    Owner

    Good question! Let's break it down, here's the pipeline:

    1. Get a feed of images from the camera > 2. Input the images into MLKit (this library) > 3. augment the preview screen with OCR data.

    Seems like https://github.com/tensorflow/tfjs/blob/d8a8afeeb9218e39655712c3d8d26977371054bf/tfjs-react-native/src/camera/camera_stream.tsx#L369 would be a good reference for 1 and 3.

    I would start with 1 and 3 - i.e. to make sure that I am able to receive a feed of images and augment them (with a static text). 1 would produce a stream of in-memory, platform-specific abstractions of "image". We'll need to feed those images into mlkit. This library need an enhancement - to be able to receive the "image" object instead of a file URL. It is already supported in native packages, we'll just need to expose those methods to RN.

  2. Kommandat commented on Sep 28, 2021

    @Kommandat
    Author

    Thanks for outlining the steps! I don't have time to do this now, but I'm definitely interested in tackling this and making the enhancement to this library in ~1 month or so.

    A few questions:

    • The Tensorflow Camera Stream returns an iterable of Tensor3Ds. How would this be converted into a CMSampleBufferRef or ByteBuffer / ByteArray that MLKit needs? Do you think Tensor3D.buffer() is sufficient?
    • Which files in this library should I familiarize myself with to understand what would be needed for this enhancement?
    • I'm not too familiar with how to integrate native code with React native (so far I've just been working with React Native & Expo, using libraries like this one when I need native functionality). Do you have any recommended resources to learn more on this topic?

    Thanks!

  3. Noitidart commented on Sep 28, 2021

    @Noitidart

    Thanks all for sharing this, this is complicated and I was also trying frame processor, but was having a tough time.

  4. agoldis commented on Sep 29, 2021

    @agoldis
    Owner
    • The Tensorflow Camera Stream returns an iterable of Tensor3Ds. How would this be converted into a CMSampleBufferRef or ByteBuffer / ByteArray that MLKit needs? Do you think Tensor3D.buffer() is sufficient?

    I am not too familiar with Tensor and its terms. The referenced file https://github.com/tensorflow/tfjs/blob/d8a8afeeb9218e39655712c3d8d26977371054bf/tfjs-react-native/src/camera/camera_stream.tsx#L291 integrates with Expo Camera package. It "renders" two React elements;

    return [cameraComp, glViewComponent];
    

    The first one "starts" expo camera, the seconds one shows a GLView that render augmented camera output. Seems like all the magic is happening in nextFrameGenerator.

    Replacing Tensor logic with mlkit logic would produce the desired effect I think.

    Which files in this library should I familiarize myself with to understand what would be needed for this enhancement?

    I'm not too familiar with how to integrate native code with React native (so far I've just been working with React Native & Expo, using libraries like this one when I need native functionality). Do you have any recommended resources to learn more on this topic?

    Just follow the official RN guide - it's quite a detailed documentation and this package is very "usual" in terms of implementation - it's just one class for each platform.

  5. Repository owner locked and limited conversation to collaborators on Sep 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions