What's the output of an Embedding layer in tensorflow and what does GlobalAveragePooling1D do it?

Solution 1:

To your first question: What's the output of an Embedding layer in tensorflow?

The Embedding layer maps each integer value in a sequence that represents a unique word in the vocabulary to a 7-dimensional vector. In the following example, you have two sequences with 10 integer values each. These integer values can range from 0 to 29, where 30 is the size of the vocabulary. Each integer value of each sequence is mapped to a 7-dimensional vector, resulting in the output shape (2, 10, 7), where 2 is the number of samples, 10 is the sequence length and 7 is the dimension of each integer value:

import tensorflow as tf

samples = 2
texts = tf.random.uniform((samples, 10), maxval=30, dtype=tf.int32)

embedding_layer = tf.keras.layers.Embedding(30, 7, input_length=10)
print(embedding_layer(texts))

tf.Tensor(
[[[ 0.0225671   0.02347589  0.00979777  0.00041901 -0.00628462
    0.02810872 -0.00962182]
  [-0.00848696 -0.04342243 -0.02836052 -0.00517335 -0.0061365
   -0.03012114  0.01677728]
  [ 0.03311044  0.00556745 -0.00702027  0.03381392 -0.04623893
    0.04987461 -0.04816799]
  [-0.03521906  0.0379228   0.03005264 -0.0020758  -0.0384485
    0.04822161 -0.02092661]
  [-0.03521906  0.0379228   0.03005264 -0.0020758  -0.0384485
    0.04822161 -0.02092661]
  [-0.01790254 -0.0175228  -0.01194855 -0.02171307 -0.0059397
    0.02812174  0.01709754]
  [ 0.03117083  0.03501941  0.01058724  0.0452967  -0.03717183
   -0.04691924  0.04459465]
  [-0.0225444   0.01631368 -0.04825303  0.02976335  0.03874404
    0.01886607 -0.04535152]
  [-0.01405543 -0.01035894 -0.01828993  0.01214089 -0.0163126
    0.00249451 -0.03320551]
  [-0.00536104  0.04976835  0.03676006 -0.04985759 -0.04882429
    0.04079831 -0.04694915]]

 [[ 0.02474061  0.04651412  0.01263839  0.02834389  0.01770737
    0.027616    0.0391163 ]
  [-0.00848696 -0.04342243 -0.02836052 -0.00517335 -0.0061365
   -0.03012114  0.01677728]
  [-0.02423838  0.00046005  0.01264722 -0.00118362 -0.04956226
   -0.00222496  0.00678415]
  [ 0.02132202  0.02490019  0.015528    0.01769954  0.03830704
   -0.03469712 -0.00817447]
  [-0.03713315 -0.01064591  0.0106518  -0.00899752 -0.04772154
    0.03767705 -0.02580358]
  [ 0.02132202  0.02490019  0.015528    0.01769954  0.03830704
   -0.03469712 -0.00817447]
  [ 0.00416059 -0.03158562  0.00862025 -0.03387908  0.02394537
   -0.00088609  0.01963869]
  [-0.0454465   0.03087567 -0.01201812 -0.02580545  0.02585572
   -0.00974055 -0.02253721]
  [-0.00438716  0.03688161  0.04575384 -0.01561296 -0.0137012
   -0.00927494 -0.02183568]
  [ 0.0225671   0.02347589  0.00979777  0.00041901 -0.00628462
    0.02810872 -0.00962182]]], shape=(2, 10, 7), dtype=float32)

When working with text data, the output of an Embedding layer would be 2 sentences consisting of 10 words each, where each word is mapped to a 7-dimensional vector.

If you are wondering where these random numbers for each integer in each sequence come from, by default the Embedding layer uses a uniform distribution to generate these values.

To your second question: What does a 1D global average pooling do to an Embedding layer?

The layer GlobalAveragePooling1D does nothing more than simply calculate the average over a given dimension in a tensor. The following example calculates the average of the 7 numbers representing a word in each sequence and returns a scalar for each word, resulting in the output shape (2, 10), where 2 is the number of samples (sentences) and 10 represents the average values for of each word. This is equivalent to simply doing tf.reduce_mean(embedding_layer(texts), axis=-1).

import tensorflow as tf

samples = 2
texts = tf.random.uniform((samples, 10), maxval=30, dtype=tf.int32)

embedding_layer = tf.keras.layers.Embedding(30, 7, input_length=10)
average_layer = tf.keras.layers.GlobalAveragePooling1D(data_format = "channels_first")
print(average_layer(embedding_layer(texts)))