加载 BertModel 时出现警告
tensorflow 1496
原文标题 ：Warning while loading the BertModel
我使用以下代码片段加载 BERT 模型：
name = "bert-base-uncased"
from transformers import BertModel
from transformers import BertTokenizer
print("[ Using pretrained BERT embeddings ]")
self.bert_tokenizer = BertTokenizer.from_pretrained(name, do_lower_case=lower_case)
self.bert_model = BertModel.from_pretrained(name)
if fix_emb:
print("[ Fix BERT layers ]")
self.bert_model.eval()
for param in self.bert_model.parameters():
param.requires_grad = False
else:
print("[ Finetune BERT layers ]")
self.bert_model.train()
但我收到以下错误：
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
这里出了什么问题？
回复
我来回复-
Yongle Liu 评论这些可能会有所帮助。
您正在 BertForSequenceClassification 模型中加载基于 bert 的检查点（这是一个使用与 BertForPreTraining 类似的架构进行训练的检查点）。
这意味着：
The layers that BertForPreTraining has, but BertForSequenceClassification does not have will be discarded The layers that BertForSequenceClassification has but BertForPreTraining does not have will be randomly initialized.这是意料之中的，并告诉您在微调之前，您的 BertForSequenceClassification 模型不会有良好的性能。
2年前 0条评论
