如何优化我的代码以逆变换 TextVectorization 的输出？

青葱年少 nlp 202

原文标题 ：How can I optimize my code to inverse transform the output of TextVectorization?

我在 TF Keras Sequential 模型中使用 TextVectorization 层。我需要将中间 TextVectorization 层的输出转换为纯文本。我发现没有直接的方法可以做到这一点。所以我使用 TextVectorization 层的词汇对向量进行逆变换。代码如下：

from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
text_list = np.array(["this is the first sentence.","second line of the dataset."]) # a list of 2 sentences
textvectorizer = TextVectorization(max_tokens=None,
            standardize=None,
            ngrams=None,
            output_mode="int",
            output_sequence_length=None,
            pad_to_max_tokens=False)
textvectorizer.adapt(text_list)
vectors = textvectorizer(text_list)
vectors

矢量图：

array([[ 3,  7,  2,  9,  4],
       [ 5,  6,  8,  2, 10]])

现在，我想将向量转换为句子。

my_vocab = textvectorizer.get_vocabulary()
plain_text_list = []
for line in vectors:
    text = ' '.join(my_vocab[idx] for idx in line)
    plain_text_list.append(text)

print(plain_text_list)

输出：

['this is the first sentence.', 'second line of the dataset.']

我成功地获得了预期的结果。但是，由于我在代码中使用的幼稚方法，当应用于大型数据集时，这种方法非常慢。我想减少这个方法的执行时间。

原文链接：https://stackoverflow.com//questions/71496947/how-can-i-optimize-my-code-to-inverse-transform-the-output-of-textvectorization

回复

我来回复
  • AloneTogether的头像
    AloneTogether 评论

    也许试试np.vectorize：

    import numpy as np

my_vocab = textvectorizer.get_vocabulary()
index_vocab =  dict(zip(np.arange(len(my_vocab)), my_vocab))
print(np.vectorize(index_vocab.get)(vectors))

    [['this' 'is' 'the' 'first' 'sentence.']
 ['second' 'line' 'of' 'the' 'dataset.']]

    性能测试：

    import numpy as np
import timeit

my_vocab = textvectorizer.get_vocabulary()

def method1(my_vocab, vectors):
  index_vocab =  dict(zip(np.arange(len(my_vocab)), my_vocab))
  return np.vectorize(index_vocab.get)(vectors)

def method2(my_vocab, vectors):
  plain_text_list = []
  for line in vectors:
      text = ' '.join(my_vocab[idx] for idx in line)
      plain_text_list.append(text)
  return plain_text_list

t1 = timeit.Timer(lambda: method1(my_vocab, vectors))
t2 = timeit.Timer(lambda: method2(my_vocab, vectors)) 

print(t1.timeit(5000))
print(t2.timeit(5000))

    0.3139600929998778
19.671524284000043
    2年前 0条评论

相关问题