The ML.DECODE_IMAGE function

This document describes the ML.DECODE_IMAGE scalar function, which lets you convert image bytes to a multi-dimensional ARRAY representation. This conversion lets you do inference using the ML.PREDICT function on vision models that require images to be in ARRAY format.

Syntax

 ML.DECODE_IMAGE(data) AS input_field

Arguments

ML.DECODE_IMAGE takes the following argument:

  • data: this argument must be a reference to one of the following:

    • A BYTES value that represents an image in one of the following formats:

      • JPEG
      • PNG
      • BMP

    • The data pseudocolumn of an object table.

      If you are referencing only the object table in the query_statement of the ML.PREDICT function, you can specify data as the value. If you are joining the object table with other tables in the query_statement of the ML.PREDICT function, then you should fully qualify the data value as dataset_name.object_table_name.data. This lets you avoid ambiguity if the tables share any column names.

    The data argument value must be <= 10 MB (10,000,000 bytes).

Output

ML.DECODE_IMAGE returns a STRUCT<ARRAY<INT64>, ARRAY<FLOAT64>> value that represents the image.

The first array in the struct represents the dimensions of the image. It contains three INT64 values, which represent the image height (H), width (W), and number of channels (C).

The second array in the struct contains the decoded image data. The length of the array is equivalent to H x W x C from the preceding array. Each value in the array is between [0, 1).

For example, a 1x2 image in the RGB color space might have a form similar to [[0.1,0.2,0.3],[0.4,0.5,0.6]], which would be represented as a struct in the form ([1,2,3],[0.1,0.2,0.3,0.4,0.5,0.6]).

If the struct value is larger than 60 MB, it is downscaled to that size while preserving aspect ratio.

You can use ML.DECODE_IMAGE output directly in an ML.PREDICT function, or you can write the results to a table column and reference that column when you call ML.PREDICT. You can also pass ML.DECODE_IMAGE output to another image processing function for additional preprocessing.

If you are using ML.DECODE_IMAGE output directly in an ML.PREDICT function, you must alias the output of this function with the name of an input field for the model that is referenced in the ML.PREDICT function. You can find this information by inspecting the model and looking at the field names in the Features section. For more information, see Run inference on image object tables.

Examples

The following examples show different ways you can use the ML.DECODE_IMAGE function.

Example 1

The following example uses the ML.DECODE_IMAGE function directly in the ML.PREDICT function. It returns the inference results for all images in the object table, for a model with an input field of input and an output field of feature:

SELECT * FROM ML.PREDICT(   MODEL `my_dataset.vision_model`,   (SELECT uri, ML.RESIZE_IMAGE(ML.DECODE_IMAGE(data), 480, 480, FALSE) AS input   FROM `my_dataset.object_table`) );

Example 2

The following example uses the ML.DECODE_IMAGE function directly in the ML.PREDICT function, and uses the ML.CONVERT_COLOR_SPACE function in the ML.PREDICT function to convert the image color space from RBG to YIQ. It also shows how to use object table fields to filter the objects included in inference. It returns the inference results for all JPG images in the object table, for a model with an input field of input and an output field of feature:

SELECT * FROM   ML.PREDICT(     MODEL `my_dataset.vision_model`,     (SELECT uri, ML.CONVERT_COLOR_SPACE(ML.RESIZE_IMAGE(ML.DECODE_IMAGE(data), 224, 280, TRUE), 'YIQ') AS input     FROM `my_dataset.object_table`     WHERE content_type = 'image/jpeg')   );

Example 3

The following example uses results from ML.DECODE_IMAGE that have been written to a table column but not processed any further. It uses ML.RESIZE_IMAGE and ML.CONVERT_IMAGE_TYPE in the ML.PREDICT function to process the image data. It returns the inference results for all images in the decoded images table, for a model with an input field of input and an output field of feature.

Create the decoded images table:

CREATE OR REPLACE TABLE `my_dataset.decoded_images`   AS (SELECT ML.DECODE_IMAGE(data) AS decoded_image   FROM `my_dataset.object_table`);

Run inference on the decoded images table:

SELECT * FROM ML.PREDICT(   MODEL`my_dataset.vision_model`,   (SELECT uri, ML.CONVERT_IMAGE_TYPE(ML.RESIZE_IMAGE(decoded_image, 480, 480, FALSE)) AS input   FROM `my_dataset.decoded_images`) );

Example 4

The following example uses results from ML.DECODE_IMAGE that have been written to a table column and preprocessed using ML.RESIZE_IMAGE. It returns the inference results for all images in the decoded images table, for a model with an input field of input and an output field of feature.

Create the table:

CREATE OR REPLACE TABLE `my_dataset.decoded_images`   AS (SELECT ML.RESIZE_IMAGE(ML.DECODE_IMAGE(data) 480, 480, FALSE) AS decoded_image   FROM `my_dataset.object_table`);

Run inference on the decoded images table:

SELECT * FROM ML.PREDICT(   MODEL `my_dataset.vision_model`,   (SELECT uri, decoded_image AS input   FROM `my_dataset.decoded_images`) );

Example 5

The following example uses the ML.DECODE_IMAGE function directly in the ML.PREDICT function. In this example, the model has an output field of embeddings and two input fields: one that expects an image, f_img, and one that expects a string, f_txt. The image input comes from the object table and the string input comes from a standard BigQuery table that is joined with the object table by using the uri column.

SELECT * FROM   ML.PREDICT(     MODEL `my_dataset.mixed_model`,     (SELECT uri, ML.RESIZE_IMAGE(ML.DECODE_IMAGE(my_dataset.my_object_table.data), 224, 224, FALSE) AS f_img,       my_dataset.image_description.description AS f_txt     FROM `my_dataset.object_table`     JOIN `my_dataset.image_description`     ON object_table.uri = image_description.uri)   );

What's next