OAG BERT

库|纸|松弛

我们在COGDL软件包中发布了两个版本的Oag-Bert（由于先前的Tsinghua云磁盘下载到期，请立即从ModelsCope手动下载模型）。 OAG-BERT是一种异质实体的学术语言模型，不仅了解OAG中的学术文本，而且了解异质的实体知识。加入我们的Slack或Google Group以获取任何评论和请求！我们的论文在这里。

V1：香草版本

基本版本Oag-Bert。与Scibert类似，我们在开放学术图中预先培训BERT模型，包括纸张标题，摘要和身体。

Oag-Bert的使用是普通Scibert或Bert的。例如，您可以使用以下代码编码两个文本序列并检索其输出

 from cogdl . oag import oagbert

tokenizer , bert_model = oagbert ()

sequence = [ "CogDL is developed by KEG, Tsinghua." , "OAGBert is developed by KEG, Tsinghua." ]
tokens = tokenizer ( sequence , return_tensors = "pt" , padding = True )
outputs = bert_model ( ** tokens )

V2：实体增强版本

香草Oag-Bert的扩展。我们将丰富的实体信息纳入开放式学术图中，例如作者和研究领域。因此，您可以在Oag-Bert V2中编码各种类型的实体。例如，要编码伯特的论文，您可以使用以下代码

 from cogdl . oag import oagbert
import torch

tokenizer , model = oagbert ( "oagbert-v2" )
title = 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding'
abstract = 'We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation...'
authors = [ 'Jacob Devlin' , 'Ming-Wei Chang' , 'Kenton Lee' , 'Kristina Toutanova' ]
venue = 'north american chapter of the association for computational linguistics'
affiliations = [ 'Google' ]
concepts = [ 'language model' , 'natural language inference' , 'question answering' ]
# build model inputs
input_ids , input_masks , token_type_ids , masked_lm_labels , position_ids , position_ids_second , masked_positions , num_spans = model . build_inputs (
    title = title , abstract = abstract , venue = venue , authors = authors , concepts = concepts , affiliations = affiliations
)
# run forward
sequence_output , pooled_output = model . bert . forward (
    input_ids = torch . LongTensor ( input_ids ). unsqueeze ( 0 ),
    token_type_ids = torch . LongTensor ( token_type_ids ). unsqueeze ( 0 ),
    attention_mask = torch . LongTensor ( input_masks ). unsqueeze ( 0 ),
    output_all_encoded_layers = False ,
    checkpoint_activations = False ,
    position_ids = torch . LongTensor ( position_ids ). unsqueeze ( 0 ),
    position_ids_second = torch . LongTensor ( position_ids_second ). unsqueeze ( 0 )
)

您还可以使用一些集成功能直接使用OAG-BERT V2，例如使用decode_beamsearch基于现有上下文来生成实体。例如，要用2个令牌生成概念，请运行以下代码

 model . eval ()
candidates = model . decode_beamsearch (
    title = title ,
    abstract = abstract ,
    venue = venue ,
    authors = authors ,
    affiliations = affiliations ,
    decode_span_type = 'FOS' ,
    decode_span_length = 2 ,
    beam_width = 8 ,
    force_forward = False
)

Oag-Bert在广泛的实体感知任务上超越了其他学术语言模型，同时保持其在普通NLP任务上的性能。

超过

我们还为用户发布了另外两个V2版本。

一个是基于一代的版本，可用于基于其他信息生成文本。例如，使用以下代码自动生成带有摘要的纸质标题。

 from cogdl . oag import oagbert

tokenizer , model = oagbert ( 'oagbert-v2-lm' )
model . eval ()

for seq , prob in model . generate_title ( abstract = "To enrich language models with domain knowledge is crucial but difficult. Based on the world's largest public academic graph Open Academic Graph (OAG), we pre-train an academic language model, namely OAG-BERT, which integrates massive heterogeneous entities including paper, author, concept, venue, and affiliation. To better endow OAG-BERT with the ability to capture entity information, we develop novel pre-training strategies including heterogeneous entity type embedding, entity-aware 2D positional encoding, and span-aware entity masking. For zero-shot inference, we design a special decoding strategy to allow OAG-BERT to generate entity names from scratch. We evaluate the OAG-BERT on various downstream academic tasks, including NLP benchmarks, zero-shot entity inference, heterogeneous graph link prediction, and author name disambiguation. Results demonstrate the effectiveness of the proposed pre-training approach to both comprehending academic texts and modeling knowledge from heterogeneous entities. OAG-BERT has been deployed to multiple real-world applications, such as reviewer recommendations for NSFC (National Nature Science Foundation of China) and paper tagging in the AMiner system. It is also available to the public through the CogDL package." ):
    print ( 'Title: %s' % seq )
    print ( 'Perplexity: %.4f' % prob )
# One of our generations: "pre-training oag-bert: an academic language model for enriching academic texts with domain knowledge"

除此之外，我们还根据名称歧义任务来计算纸张相似性，以计算Oag-Bert，该任务被称为句子 - 奥格伯特之后。以下代码展示了使用句子 - 奥格伯特计算纸张相似性的示例。

 import os
from cogdl . oag import oagbert
import torch
import torch . nn . functional as F
import numpy as np


# load time
tokenizer , model = oagbert ( "oagbert-v2-sim" )
model . eval ()

# Paper 1
title = 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding'
abstract = 'We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation...'
authors = [ 'Jacob Devlin' , 'Ming-Wei Chang' , 'Kenton Lee' , 'Kristina Toutanova' ]
venue = 'north american chapter of the association for computational linguistics'
affiliations = [ 'Google' ]
concepts = [ 'language model' , 'natural language inference' , 'question answering' ]

# encode first paper
input_ids , input_masks , token_type_ids , masked_lm_labels , position_ids , position_ids_second , masked_positions , num_spans = model . build_inputs (
    title = title , abstract = abstract , venue = venue , authors = authors , concepts = concepts , affiliations = affiliations
)
_ , paper_embed_1 = model . bert . forward (
    input_ids = torch . LongTensor ( input_ids ). unsqueeze ( 0 ),
    token_type_ids = torch . LongTensor ( token_type_ids ). unsqueeze ( 0 ),
    attention_mask = torch . LongTensor ( input_masks ). unsqueeze ( 0 ),
    output_all_encoded_layers = False ,
    checkpoint_activations = False ,
    position_ids = torch . LongTensor ( position_ids ). unsqueeze ( 0 ),
    position_ids_second = torch . LongTensor ( position_ids_second ). unsqueeze ( 0 )
)

# Positive Paper 2
title = 'Attention Is All You Need'
abstract = 'We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely...'
authors = [ 'Ashish Vaswani' , 'Noam Shazeer' , 'Niki Parmar' , 'Jakob Uszkoreit' ]
venue = 'neural information processing systems'
affiliations = [ 'Google' ]
concepts = [ 'machine translation' , 'computation and language' , 'language model' ]

input_ids , input_masks , token_type_ids , masked_lm_labels , position_ids , position_ids_second , masked_positions , num_spans = model . build_inputs (
    title = title , abstract = abstract , venue = venue , authors = authors , concepts = concepts , affiliations = affiliations
)
# encode second paper
_ , paper_embed_2 = model . bert . forward (
    input_ids = torch . LongTensor ( input_ids ). unsqueeze ( 0 ),
    token_type_ids = torch . LongTensor ( token_type_ids ). unsqueeze ( 0 ),
    attention_mask = torch . LongTensor ( input_masks ). unsqueeze ( 0 ),
    output_all_encoded_layers = False ,
    checkpoint_activations = False ,
    position_ids = torch . LongTensor ( position_ids ). unsqueeze ( 0 ),
    position_ids_second = torch . LongTensor ( position_ids_second ). unsqueeze ( 0 )
)

# Negative Paper 3
title = "Traceability and international comparison of ultraviolet irradiance"
abstract = "NIM took part in the CIPM Key Comparison of ″Spectral Irradiance 250 to 2500 nm″. In UV and NIR wavelength, the international comparison results showed that the consistency between Chinese value and the international reference one"
authors =  [ 'Jing Yu' , 'Bo Huang' , 'Jia-Lin Yu' , 'Yan-Dong Lin' , 'Cai-Hong Dai' ]
veune = 'Jiliang Xuebao/Acta Metrologica Sinica'
affiliations = [ 'Department of Electronic Engineering' ]
concept = [ 'Optical Division' ]

input_ids , input_masks , token_type_ids , masked_lm_labels , position_ids , position_ids_second , masked_positions , num_spans = model . build_inputs (
    title = title , abstract = abstract , venue = venue , authors = authors , concepts = concepts , affiliations = affiliations
)
# encode thrid paper
_ , paper_embed_3 = model . bert . forward (
    input_ids = torch . LongTensor ( input_ids ). unsqueeze ( 0 ),
    token_type_ids = torch . LongTensor ( token_type_ids ). unsqueeze ( 0 ),
    attention_mask = torch . LongTensor ( input_masks ). unsqueeze ( 0 ),
    output_all_encoded_layers = False ,
    checkpoint_activations = False ,
    position_ids = torch . LongTensor ( position_ids ). unsqueeze ( 0 ),
    position_ids_second = torch . LongTensor ( position_ids_second ). unsqueeze ( 0 )
)

# calulate text similarity
# normalize
paper_embed_1 = F . normalize ( paper_embed_1 , p = 2 , dim = 1 )
paper_embed_2 = F . normalize ( paper_embed_2 , p = 2 , dim = 1 )
paper_embed_3 = F . normalize ( paper_embed_3 , p = 2 , dim = 1 )

# cosine sim.
sim12 = torch . mm ( paper_embed_1 , paper_embed_2 . transpose ( 0 , 1 ))
sim13 = torch . mm ( paper_embed_1 , paper_embed_3 . transpose ( 0 , 1 ))
print ( sim12 , sim13 )

这种微调是关于Whoisho Who Name Digammuation任务的。同一作者撰写的论文被视为正面，其余的是负对。我们采样了0.4m的阳性对和1.6m负对，并使用约束学习来微调Oag-Bert（版本2）。对于50％的情况，我们仅使用纸质标题，而其他50％使用所有异质信息。我们使用平均互惠等级评估绩效，其中较高的值表明结果更好。测试集的性能如下所示。

	Oagbert-V2	Oagbert-V2-SIM
标题	0.349	0.725
标题+摘要+作者+AFF+场地	0.355	0.789

有关更多详细信息，请参阅cogdl中的示例/oagbert_metainfo.py。

中文版本

我们还培训了中国的Oagbert供使用。该模型已在包括4400万份纸元数据在内的语料库中进行了预培训，包括标题，摘要，作者，分支机构，场地，关键字和资金。新实体基金的扩展超出了英语版本中使用的实体。此外，中国奥格伯特接受了句子令牌训练。这是英国Oagbert和中国Oagbert之间的两个主要区别。

可以在示例/oagbert/oagbert_metainfo_zh.py和示例中找到使用原始中国oagbert和句子 - 奥格伯特的示例。与英语句子 - 奥格伯特类似，中文句子 - 奥格伯特的名称歧义任务是微调的，用于计算纸张嵌入相似性。性能如下所示。如果下游任务没有足够的数据进行微调，我们建议用户直接使用此版本。

	Oagbert-V2-ZH	OAGBERT-V2-ZH-SIM
标题	0.337	0.619
标题+摘要	0.314	0.682

引用

如果您发现它有用，请在您的工作中引用我们：

 @article{xiao2021oag,
  title={OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Model},
  author={Liu, Xiao and Yin, Da and Zhang, Xingjian and Su, Kai and Wu, Kan and Yang, Hongxia and Tang, Jie},
  journal={arXiv preprint arXiv:2103.02410},
  year={2021}
}
@inproceedings{zhang2019oag,
  title={OAG: Toward Linking Large-scale Heterogeneous Entity Graphs.},
  author={Zhang, Fanjin and Liu, Xiao and Tang, Jie and Dong, Yuxiao and Yao, Peiran and Zhang, Jie and Gu, Xiaotao and Wang, Yan and Shao, Bin and Li, Rui and Wang, Kuansan},
  booktitle={Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’19)},
  year={2019}
}
@article{chen2020conna,
  title={CONNA: Addressing Name Disambiguation on The Fly},
  author={Chen, Bo and Zhang, Jing and Tang, Jie and Cai, Lingfan and Wang, Zhaoyu and Zhao, Shu and Chen, Hong and Li, Cuiping},
  journal={IEEE Transactions on Knowledge and Data Engineering},
  year={2020},
  publisher={IEEE}
}