You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Dec 9, 2025. It is now read-only.
Repository navigation
This repository was archived by the owner on Dec 9, 2025. It is now read-only.
Question: How to perform OCR on Camera's live frame data? #16
Really more of a question (or a feature request), but is it possible to perform OCR on the live frame data from the Camera?
In my app I was trying to get around this by repeatedly taking pictures at regular intervals and scanning them to show red bounding boxes for the text (video below). This would've worked fine, except iOS makes a loud shutter noise when you take a picture (and a shutter noise every 500ms is really really annoying).
It looks like expo-camera (which is what I'm using) doesn't access the live frame data but there are some other options out there that do, but I don't know how to get them to work with MLKit:
Thanks! Appreciate this package a ton and understand what I'm asking here might be difficult to do. If so, I'm also curious to learn more and help out.
I would start with 1 and 3 - i.e. to make sure that I am able to receive a feed of images and augment them (with a static text). 1 would produce a stream of in-memory, platform-specific abstractions of "image". We'll need to feed those images into mlkit. This library need an enhancement - to be able to receive the "image" object instead of a file URL. It is already supported in native packages, we'll just need to expose those methods to RN.
Thanks for outlining the steps! I don't have time to do this now, but I'm definitely interested in tackling this and making the enhancement to this library in ~1 month or so.
A few questions:
The Tensorflow Camera Stream returns an iterable of Tensor3Ds. How would this be converted into a CMSampleBufferRef or ByteBuffer / ByteArray that MLKit needs? Do you think Tensor3D.buffer() is sufficient?
Which files in this library should I familiarize myself with to understand what would be needed for this enhancement?
I'm not too familiar with how to integrate native code with React native (so far I've just been working with React Native & Expo, using libraries like this one when I need native functionality). Do you have any recommended resources to learn more on this topic?
The Tensorflow Camera Stream returns an iterable of Tensor3Ds. How would this be converted into a CMSampleBufferRef or ByteBuffer / ByteArray that MLKit needs? Do you think Tensor3D.buffer() is sufficient?
The first one "starts" expo camera, the seconds one shows a GLView that render augmented camera output. Seems like all the magic is happening in nextFrameGenerator.
Replacing Tensor logic with mlkit logic would produce the desired effect I think.
Which files in this library should I familiarize myself with to understand what would be needed for this enhancement?
I'm not too familiar with how to integrate native code with React native (so far I've just been working with React Native & Expo, using libraries like this one when I need native functionality). Do you have any recommended resources to learn more on this topic?
Just follow the official RN guide - it's quite a detailed documentation and this package is very "usual" in terms of implementation - it's just one class for each platform.
Repository owner locked and limited conversation to collaborators on Sep 29, 2021
Activity
agoldis commentedon Sep 25, 2021
Good question! Let's break it down, here's the pipeline:
Seems like https://github.com/tensorflow/tfjs/blob/d8a8afeeb9218e39655712c3d8d26977371054bf/tfjs-react-native/src/camera/camera_stream.tsx#L369 would be a good reference for 1 and 3.
I would start with 1 and 3 - i.e. to make sure that I am able to receive a feed of images and augment them (with a static text). 1 would produce a stream of in-memory, platform-specific abstractions of "image". We'll need to feed those images into mlkit. This library need an enhancement - to be able to receive the "image" object instead of a file URL. It is already supported in native packages, we'll just need to expose those methods to RN.
Kommandat commentedon Sep 28, 2021
Thanks for outlining the steps! I don't have time to do this now, but I'm definitely interested in tackling this and making the enhancement to this library in ~1 month or so.
A few questions:
CMSampleBufferReforByteBuffer/ByteArraythat MLKit needs? Do you think Tensor3D.buffer() is sufficient?Thanks!
Noitidart commentedon Sep 28, 2021
Thanks all for sharing this, this is complicated and I was also trying frame processor, but was having a tough time.
agoldis commentedon Sep 29, 2021
I am not too familiar with Tensor and its terms. The referenced file https://github.com/tensorflow/tfjs/blob/d8a8afeeb9218e39655712c3d8d26977371054bf/tfjs-react-native/src/camera/camera_stream.tsx#L291 integrates with Expo Camera package. It "renders" two React elements;
The first one "starts" expo camera, the seconds one shows a
GLViewthat render augmented camera output. Seems like all the magic is happening innextFrameGenerator.Replacing Tensor logic with mlkit logic would produce the desired effect I think.
Just follow the official RN guide - it's quite a detailed documentation and this package is very "usual" in terms of implementation - it's just one class for each platform.