Category ""
ADVERTISING
size mismatch for vision_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([197, 768]) from checkpoint, the shape in current model is torch.Size([50, 768]). Environment info
When I load the pre-trained weights from ViT-base-patch16-384 (available on huggingface) to my model using load_state_dict(), I am getting the following error: size mismatch for pos_embed: copying a param with shape torch.Size ( [1, 577, 768]) from checkpoint, the shape in current model is torch.Size ( [1, 1601, 768]).
size mismatch for vision_model.embeddings.patch_embedding.weight: copying a param with shape torch.Size([1152, 3, 14, 14]) from checkpoint, the shape in current model is torch.Size([1024, 3, 14, 14]).
You can find here a solution for the error, which involves first removing the layers with mismatched sizes from the state_dict and only then loading it with strict=False. In your case, it would look like this: It seems to me that your model configuration does not match the content of the model checkpoint.
size mismatch for vision_model.encoder.layers.0.self_attn.k_proj.weight: copying a param with shape torch.Size([1152, 1152]) from checkpoint, the shape in current model is torch.Size([768, 768]). The text was updated successfully, but these errors were encountered:
size mismatch for text_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([77, 768]) from checkpoint, the shape in current model is torch.Size([77, 512]).
As you have already mentioned, you can use ignore_mismatched_sizes to load your model. This parameter will load the the embedding and encoding layers of your model, but will randomly initialize the classification head:
Optimize the model hyper-parameters using the Ray tune method. Embedding size mismatch occurs as follows. I've added two special tokens (e.g., [e], [/e]), and I got this error.
EncoderDecoderModel.from_pretrained fails to load the model when the configuration is loaded from the previously checkpointed model. I believe it's because it is loading a default vocab size (30522) instead of whatever is defined in the saved config (119547) in my case.
I would like to ask about the way to change the embedding size of the trained model. I have a trained model models/BERT-pretrain-1-step-5000.pkl . Now I am adding a new token [TRA] to the tokeniser and try to use the resize_token_embeddings to the pertained one.