论文背景

标题：SPECTER: Document-level Representation Learning using Citation-informed Transformers
摘要：表示学习是自然语言处理系统的关键组成部分。像BERT这样的最新Transformer语言模型学习了强大的文本表示，但这些模型针对标记和句子级别的训练目标，并不利用相关性信息，这限制了它们在文档级表示方面的能力。对于科学文献的应用，如分类和推荐，嵌入提供了强大的终端任务性能。我们提出了SPECTER，一种基于预训练Transformer语言模型的科学文档的文档级嵌入生成方法，其依据一个强大的文档级相关性信号：引用图。与现有的预训练语言模型不同，SPECTER可以轻松地应用于下游应用，而无需特定任务的微调。此外，为了鼓励进一步的文档级模型研究，我们引入了SciDocs，一个新的评估基准，包括七个文档级任务，从引文预测到文档分类和推荐。我们展示了SPECTER在基准测试上优于各种竞争基线。
作者：【ArmanAI】Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld
CCF：B
会议：Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 2020

模型细节

文档：https://github.com/allenai/specter#specter-document-level-representation-learning-using-citation-informed-transformers

安装Transformers

Python 3.6+, PyTorch 1.1.0+, TensorFlow 2.0+, and Flax.

通过Github仓库安装1. 下载

## git clone git@github.com:allenai/specter.gitgit clone https://github.com/allenai/specter.gitcd specter## 通过浏览器下载archive.tar.gz更快wget https://ai2-s2-research-public.s3-us-west-2.amazonaws.com/specter/archive.tar.gztar -xzvf archive.tar.gz

2. 安装环境

## 先安装conda，记住运行Anaconda.sh安装脚本时不要在root下运行，不然默认装到/root下，虽然在安装时选择其他路径可以让user使用，但是运行代码还是有可能出错，保险起见，就user下安装即可。conda create --name specter python=3.7 setuptools  conda activate specter  # if you don't have gpus, remove cudatoolkit argument#conda install pytorch cudatoolkit=10.1 -c pytorch   conda install pytorch cpuonly -c pytorch## pip requirements.txt的命令最好转化成以下两条，用以在网络不好的环境下分别执行，防止一个执行了重复执行## pip install dill jsonlines pandas scikit-learn##https可以换成git试试pip install -r requirements.txt  python setup.py install

3. 修改环境包依赖bug

运行代码报错，这是因为包的版本不对。
allennlp 0.9.0
overrides 7.3.1
其中overrides版本高了，应该将其改为低版本3.1.0

报错信息

(specter) user@ubuntu:~/model/model2/specter$ python scripts/embed.py \> --ids data/sample.ids --metadata data/sample-metadata.json \> --model ./model.tar.gz \> --output-file output.jsonl \> --vocab-dir data/vocab/ \> --batch-size 16 \> --cuda-device -1Traceback (most recent call last):  File "specter/predict_command.py", line 14, in     from allennlp.commands import ArgumentParserWithDefaults  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/commands/__init__.py", line 8, in     from allennlp.commands.configure import Configure  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/commands/configure.py", line 26, in     from allennlp.service.config_explorer import make_app  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/service/config_explorer.py", line 24, in     from allennlp.common.configuration import configure, choices  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/common/configuration.py", line 17, in     from allennlp.data.dataset_readers import DatasetReader  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/__init__.py", line 1, in     from allennlp.data.dataset_readers.dataset_reader import DatasetReader  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/dataset_readers/__init__.py", line 10, in     from allennlp.data.dataset_readers.ccgbank import CcgBankDatasetReader  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/dataset_readers/ccgbank.py", line 9, in     from allennlp.data.dataset_readers.dataset_reader import DatasetReader  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/dataset_readers/dataset_reader.py", line 8, in     from allennlp.data.instance import Instance  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/instance.py", line 3, in     from allennlp.data.fields.field import DataArray, Field  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/fields/__init__.py", line 7, in     from allennlp.data.fields.array_field import ArrayField  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/fields/array_field.py", line 10, in     class ArrayField(Field[numpy.ndarray]):  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/allennlp/data/fields/array_field.py", line 49, in ArrayField    @overrides  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/overrides.py", line 83, in overrides    return _overrides(method, check_signature, check_at_runtime)  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/overrides.py", line 170, in _overrides    _validate_method(method, super_class, check_signature)  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/overrides.py", line 189, in _validate_method    ensure_signature_is_compatible(super_method, method, is_static)  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/signature.py", line 102, in ensure_signature_is_compatible    ensure_return_type_compatibility(super_type_hints, sub_type_hints, method_name)  File "/home/user/anaconda3/envs/specter/lib/python3.7/site-packages/overrides/signature.py", line 303, in ensure_return_type_compatibility    f"{method_name}: return type `{sub_return}` is not a `{super_return}`."TypeError: ArrayField.empty_field: return type `None` is not a ``.

修改代码

pip install allennlp==3.1.0

4. 运行代码

此时再次运行以上代码，模型正常工作
脚本的输出大概是如下格式：