How to Label Image in Android using Firebase ML Kit?

Last Updated : 24 Feb, 2021

We have seen many apps in Android in which we will detect the object present in the image whether it may be any object. In this article, we will take a look at the implementation of image labeling in Android using Firebase ML Kit.

What we are going to build in this article?

We will be building a simple application in which we will be capturing an image of any object and from that, we will detect the objects present inside our image with the accuracy level. A sample video is given below to get an idea about what we are going to do in this article. Note that we are going to implement this project using the Java language.

Step by Step Implementation

Step 1: Create a New Project

To create a new project in Android Studio please refer to How to Create/Start a New Project in Android Studio. Note that select Java as the programming language.

Step 2: Connect your app to Firebase

After creating a new project in Android Studio connect your app to Firebase. For connecting your app to firebase. Navigate to Tools on the top bar. After that click on Firebase. A new window will open on the right side. Inside that window click on Firebase ML and then click on Use Firebase ML kit in Android. You can see the option below screenshot.

After clicking on this option on the next screen click on Connect to Firebase option to connect your app to Firebase.

Step 3: Adding dependency for language translation to build.gradle file

Navigate to the Gradle Scripts > build.gradle(Module:app) and add the below dependency in the dependencies section.

// dependency for firebase core.

implementation'com.google.firebase:firebase-core:15.0.2'

// Firebase ML dependency

implementation 'com.google.firebase:firebase-ml-vision:24.0.3'

implementation 'com.google.firebase:firebase-ml-vision-image-label-model:20.0.1'

Step 4: Adding permissions to access the Internet and meta-data in your Android Apps AndroidManifest file

Navigate to the app > AndroidManifest.xml file and add the below code to it. Comments are added in the code to get to know in more detail.

XML

<!-- below line is use to add camera feature in our app --> <uses-feature   android:name="android.hardware.camera"   android:required="true" />  <!-- permission for internet --> <uses-permission android:name="android.permission.INTERNET" />

Add the below line inside your application tag.

XML

<meta-data android:name="com.google.firebase.ml.vision.DEPENDENCIES" android:value="label" />

Below is the complete code for the AndroidManifest.xml file.

XML

<?xml version="1.0" encoding="utf-8"?> <manifest xmlns:android="http://schemas.android.com/apk/res/android"     package="com.example.labelimage">      <!-- below line is use to add camera feature in our app -->     <uses-feature         android:name="android.hardware.camera"         android:required="true" />      <!-- permission for internet -->     <uses-permission android:name="android.permission.INTERNET" />      <application         android:allowBackup="true"         android:icon="@mipmap/ic_launcher"         android:label="@string/app_name"         android:roundIcon="@mipmap/ic_launcher_round"         android:supportsRtl="true"         android:theme="@style/Theme.LabelImage">         <activity android:name=".MainActivity">             <intent-filter>                 <action android:name="android.intent.action.MAIN" />                  <category android:name="android.intent.category.LAUNCHER" />             </intent-filter>         </activity>          <meta-data             android:name="com.google.firebase.ml.vision.DEPENDENCIES"             android:value="label" />              </application>  </manifest>

Step 5: Working with the activity_main.xml file

Navigate to the app > res > layout > activity_main.xml and add the below code to that file. Below is the code for the activity_main.xml file.

XML

<?xml version="1.0" encoding="utf-8"?> <RelativeLayout      xmlns:android="http://schemas.android.com/apk/res/android"     xmlns:tools="http://schemas.android.com/tools"     android:layout_width="match_parent"     android:layout_height="match_parent"     tools:context=".MainActivity">      <!--image view for displaying our image-->     <ImageView         android:id="@+id/image"         android:layout_width="match_parent"         android:layout_height="300dp"         android:layout_alignParentTop="true"         android:layout_centerHorizontal="true"         android:layout_marginTop="10dp"         android:scaleType="centerCrop" />      <LinearLayout         android:id="@+id/idLLButtons"         android:layout_width="match_parent"         android:layout_height="wrap_content"         android:layout_below="@id/image"         android:orientation="horizontal">          <!--button for capturing our image-->         <Button             android:id="@+id/snapbtn"             android:layout_width="0dp"             android:layout_height="wrap_content"             android:layout_margin="5dp"             android:layout_marginTop="30dp"             android:layout_weight="1"             android:text="SNAP"             android:textSize="25dp"             android:textStyle="bold" />          <!--button for detecting the objects-->         <Button             android:id="@+id/labelBtn"             android:layout_width="0dp"             android:layout_height="wrap_content"             android:layout_margin="5dp"             android:layout_marginTop="30dp"             android:layout_weight="1"             android:text="Label"             android:textSize="25dp"             android:textStyle="bold" />      </LinearLayout>      <!--recycler view for displaying the list of objects-->     <androidx.recyclerview.widget.RecyclerView         android:id="@+id/idRVResults"         android:layout_width="match_parent"         android:layout_height="wrap_content"         android:layout_below="@id/idLLButtons" />  </RelativeLayout>

Step 6: Creating a modal class for storing our data

Navigate to the app > java > your app's package name > Right-click on it > New > Java class and name your class as DataModal and add the below code to it. Comments are added inside the code to understand the code in more detail.

Java

public class DataModal {      // variables for our      // string and confidence.     private String result;     private float confidence;      // constructor     public DataModal(String result, float confidence) {         this.result = result;         this.confidence = confidence;     }      // getter and setter methods     public float getConfidence() {         return confidence;     }      public void setConfidence(float confidence) {         this.confidence = confidence;     }          public String getResult() {         return result;     }      public void setResult(String result) {         this.result = result;     }      }

Step 7: Creating a layout file for displaying our recycler view items

Navigate to the app > res > layout > Right-click on it > New > layout resource file and name it as result_rv_item and add below code to it.

XML

<?xml version="1.0" encoding="utf-8"?> <androidx.cardview.widget.CardView      xmlns:android="http://schemas.android.com/apk/res/android"     xmlns:app="http://schemas.android.com/apk/res-auto"     android:layout_width="match_parent"     android:layout_height="wrap_content"     android:layout_margin="5dp"     android:elevation="8dp"     app:cardCornerRadius="8dp">      <RelativeLayout         android:layout_width="match_parent"         android:layout_height="wrap_content"         android:layout_margin="3dp">          <!--text view for our result-->         <TextView             android:id="@+id/idTVResult"             android:layout_width="match_parent"             android:layout_height="wrap_content"             android:padding="3dp"             android:text="Result"             android:textColor="@color/black" />          <!--text view for our confidence-->         <TextView             android:id="@+id/idTVConfidence"             android:layout_width="match_parent"             android:layout_height="wrap_content"             android:layout_below="@id/idTVResult"             android:padding="3dp"             android:text="Confidence"             android:textColor="@color/black" />      </RelativeLayout>  </androidx.cardview.widget.CardView>

Step 8: Creating an adapter class for our RecyclerView

Navigate to the app > java > your app's package name > Right-click on it > New > Java class and name it as resultRVAdapter and add the below code to it. Comments are added inside the code to understand the code in more detail.

Java

import android.content.Context; import android.view.LayoutInflater; import android.view.View; import android.view.ViewGroup; import android.widget.TextView;  import androidx.annotation.NonNull; import androidx.recyclerview.widget.RecyclerView;  import java.util.ArrayList;  public class resultRVAdapter extends RecyclerView.Adapter<resultRVAdapter.ViewHolder> {          // arraylist for storing our data and context     private ArrayList<DataModal> dataModalArrayList;     private Context context;      // constructor for our variables     public resultRVAdapter(ArrayList<DataModal> dataModalArrayList, Context context) {         this.dataModalArrayList = dataModalArrayList;         this.context = context;     }      @NonNull     @Override     public ViewHolder onCreateViewHolder(@NonNull ViewGroup parent, int viewType) {         // inside on create view holder method we are inflating our layout file which we created.         View view = LayoutInflater.from(parent.getContext()).inflate(R.layout.result_rv_item, parent, false);         return new ViewHolder(view);     }      @Override     public void onBindViewHolder(@NonNull ViewHolder holder, int position) {         // inside on bind view holder method we are setting         // data to each item of recycler view.         DataModal modal = dataModalArrayList.get(position);         holder.resultTV.setText(modal.getResult());         holder.confidenceTV.setText("" + modal.getConfidence());     }      @Override     public int getItemCount() {         // returning the size of array list.         return dataModalArrayList.size();     }      public class ViewHolder extends RecyclerView.ViewHolder {         // creating variables for our text view.         private TextView resultTV, confidenceTV;          public ViewHolder(@NonNull View itemView) {             super(itemView);             // initializing our views with their ids.             resultTV = itemView.findViewById(R.id.idTVResult);             confidenceTV = itemView.findViewById(R.id.idTVConfidence);         }     } }

Step 9: Working with the MainActivity.java file

Go to the MainActivity.java file and refer to the following code. Below is the code for the MainActivity.java file. Comments are added inside the code to understand the code in more detail.

Java

import android.content.Intent; import android.graphics.Bitmap; import android.os.Bundle; import android.provider.MediaStore; import android.view.View; import android.widget.Button; import android.widget.ImageView; import android.widget.Toast;  import androidx.annotation.NonNull; import androidx.appcompat.app.AppCompatActivity; import androidx.recyclerview.widget.LinearLayoutManager; import androidx.recyclerview.widget.RecyclerView;  import com.google.android.gms.tasks.OnFailureListener; import com.google.android.gms.tasks.OnSuccessListener; import com.google.firebase.ml.vision.FirebaseVision; import com.google.firebase.ml.vision.common.FirebaseVisionImage; import com.google.firebase.ml.vision.label.FirebaseVisionImageLabel; import com.google.firebase.ml.vision.label.FirebaseVisionImageLabeler;  import java.util.ArrayList; import java.util.List;  public class MainActivity extends AppCompatActivity {      // variables for our image view, image bitmap,     // buttons, recycler view, adapter and array list.     private ImageView img;     private Button snap, labelBtn;     private Bitmap imageBitmap;     private RecyclerView resultRV;     private resultRVAdapter resultRvAdapter;     private ArrayList<DataModal> dataModalArrayList;      @Override     protected void onCreate(Bundle savedInstanceState) {         super.onCreate(savedInstanceState);         setContentView(R.layout.activity_main);                  // initializing all our variables for views         img = (ImageView) findViewById(R.id.image);         snap = (Button) findViewById(R.id.snapbtn);         labelBtn = findViewById(R.id.labelBtn);         resultRV = findViewById(R.id.idRVResults);                  // initializing our array list         dataModalArrayList = new ArrayList<>();                  // initializing our adapter class.         resultRvAdapter = new resultRVAdapter(dataModalArrayList, MainActivity.this);                  // layout manager for our recycler view.         LinearLayoutManager manager = new LinearLayoutManager(MainActivity.this);                  // on below line we are setting layout manager          // and adapter to our recycler view.         resultRV.setLayoutManager(manager);         resultRV.setAdapter(resultRvAdapter);          // adding on click listener for our label button.         labelBtn.setOnClickListener(new View.OnClickListener() {             @Override             public void onClick(View v) {                 // calling method                  // to label images.                 labelImage();             }         });         // adding on click listener for our snap button.         snap.setOnClickListener(new View.OnClickListener() {             @Override             public void onClick(View v) {                 // calling a method                  // to capture an image.                 dispatchTakePictureIntent();             }         });     }      static final int REQUEST_IMAGE_CAPTURE = 1;      private void dispatchTakePictureIntent() {         // inside this method we are calling an implicit intent to capture an image.         Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);         if (takePictureIntent.resolveActivity(getPackageManager()) != null) {             // calling a start activity for result when image is captured.             startActivityForResult(takePictureIntent, REQUEST_IMAGE_CAPTURE);         }     }      @Override     protected void onActivityResult(int requestCode, int resultCode, Intent data) {         super.onActivityResult(requestCode, resultCode, data);         // inside on activity result method we are setting         // our image to our image view from bitmap.         if (requestCode == REQUEST_IMAGE_CAPTURE && resultCode == RESULT_OK) {             Bundle extras = data.getExtras();             imageBitmap = (Bitmap) extras.get("data");             // on below line we are setting our             // bitmap to our image view.             img.setImageBitmap(imageBitmap);         }     }      private void labelImage() {                // inside the label image method we are calling a          // firebase vision image and passing our image bitmap to it.         FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(imageBitmap);                  // on below line we are creating a labeler for our image bitmap         // and creating a variable for our firebase vision image labeler.         FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler();                  // calling a method to process an image and adding on success listener method to it.         labeler.processImage(image).addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {             @Override             public void onSuccess(List<FirebaseVisionImageLabel> firebaseVisionImageLabels) {                                    // inside on success method we are running a loop to get the data from our list.                 for (FirebaseVisionImageLabel label : firebaseVisionImageLabels) {                                        // on below line we are getting text from our list.                     String text = label.getText();                                          // on below line we are getting its entity id                     String entityId = label.getEntityId();                                          // on below line we are getting the                     // confidence level of our modal.                     float confidence = label.getConfidence();                                         // after getting all data we are passing it to our array list.                     dataModalArrayList.add(new DataModal(text, confidence));                                          // after adding a new data we are notifying                      // our adapter that data has been updated.                     resultRvAdapter.notifyDataSetChanged();                 }             }         }).addOnFailureListener(new OnFailureListener() {             @Override             public void onFailure(@NonNull Exception e) {                 // error handling for on failure method                 Toast.makeText(MainActivity.this, "Fail to get data..", Toast.LENGTH_SHORT).show();             }         });     } }

Now run your app and see the output of the app.

Output:

Note: If you are facing the following NDK at ~/Library/Android/sdk/ndk-bundle did not have a source.properties file error then please refer to this article.

How to Compress Image in Android Before Uploading it to Firebase Storage?

chaitanyamunje

Improve

Article Tags :

Practice Tags :

How to Label Image in Android using Firebase ML Kit?

What we are going to build in this article?

Step by Step Implementation

Output:

Similar Reads