Imports

tensor, autograd, nn_primitives

Types

Embedding[T] = object weight*: Variable[AnyTensor[T]] paddingIdx*: BiggestInt: Source Edit
EmbeddingGate[TT; scaled; Idx] {.final.} = ref object of Gate[TT]: Source Edit

Procs

proc embedding[TT; Idx: VocabIdx](input_vocab_id: Tensor[Idx]; weight: Variable[TT]; padding_idx: Idx = -1; scale_grad_by_freq: static[bool] = false): Variable[ TT]: Input:
A tensor of vocabulary indices, either:
batch_size

seq_len, batch_size

batch_size, seq_len

Vocabulary can be words, characters, series of words. Each item in your vocabulary must be encoded into an unique integer before being passed to the Embedding layer.

A weight matrix that maps those indices to the embedding vector space of shape vocabulary_size, embedding_size.

An optional padding_idx if an index corresponds to the absence of words (padding) This is necessary to support variable-length sentences. By default, the padding_idx is -1.

An optional parameter to scale the gradient by the words inverse document frequency. This divides the gradient of each words by their occurences in the minibatch. This regularise variations in the weight of very frequent words.

Source Edit
proc forward[T; Idx: VocabIdx](self: Embedding[T]; input: Tensor[Idx]): Variable[ AnyTensor[T]]: Runs input through embedding layer. Each item in your vocabulary/input must be encoded into an unique integer before being passed to the Embedding layer. Source Edit
proc init[T](ctx: Context[Tensor[T]]; layerType: typedesc[Embedding[T]]; vocabSize, embedSize: int; paddingIdx: VocabIdx = -1): Embedding[T]: Creates an embedding layer. Input:
- ``vocabSize`` Size of the vocabulary - ``embedSize`` Embedding size - ``paddingIdx`` Optional parameter for when an index corresponds to the absence of words

Returns the created Embedding.
Source Edit
func inShape[T](self: Embedding[T]): seq[int]: Source Edit
func outShape[T](self: Embedding[T]): seq[int]: Source Edit