Evaluation
Link Prediction
To assess the performance of the link prediction evaluation module of TorchKGE, it was compared with the ones of AmpliGraph (v1.3.1) and OpenKE (version of April, 9). The computation times (in seconds) reported in the following table are averaged over 5 independent evaluation processes. Experiments were done using PyTorch 1.5, TensorFlow 1.15 and a Tesla K80 GPU. Missing values for AmpliGraph are due to missing models in the library.
Model |
TransE |
TransD |
RESCAL |
ComplEx |
||||
|---|---|---|---|---|---|---|---|---|
Dataset |
FB15k |
WN18 |
FB15k |
WN18 |
FB15k |
WN18 |
FB15k |
WN18 |
AmpliGraph |
354.8 |
39.8 |
537.2 |
94.9 |
||||
OpenKE |
235.6 |
42.2 |
258.5 |
43.7 |
789.1 |
178.4 |
354.7 |
63.9 |
TorchKGE |
76.1 |
13.8 |
60.8 |
11.1 |
46.9 |
7.1 |
96.4 |
18.6 |
- class torchkge.evaluation.LinkPredictionEvaluator(model, knowledge_graph)[source]
Evaluate performance of given embedding using link prediction method.
References
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating Embeddings for Modeling Multi-relational Data. In Advances in Neural Information Processing Systems 26, pages 2787–2795, 2013. https://papers.nips.cc/paper/5071-translating-embeddings-for-modeling-multi-relational-data
- Parameters
model (torchkge.models.interfaces.Model) – Embedding model inheriting from the right interface.
knowledge_graph (torchkge.data_structures.KnowledgeGraph) – Knowledge graph on which the evaluation will be done.
- model
Embedding model inheriting from the right interface.
- kg
Knowledge graph on which the evaluation will be done.
- rank_true_heads
For each fact, this is the rank of the true head when all entities are ranked as possible replacement of the head entity. They are ranked in decreasing order of scoring function \(f_r(h,t)\).
- Type
torch.Tensor, shape: (n_facts), dtype: torch.int
- rank_true_tails
For each fact, this is the rank of the true tail when all entities are ranked as possible replacement of the tail entity. They are ranked in decreasing order of scoring function \(f_r(h,t)\).
- Type
torch.Tensor, shape: (n_facts), dtype: torch.int
- filt_rank_true_heads
This is the same as the rank_of_true_heads when in the filtered case. See referenced paper by Bordes et al. for more information.
- Type
torch.Tensor, shape: (n_facts), dtype: torch.int
- filt_rank_true_tails
This is the same as the rank_of_true_tails when in the filtered case. See referenced paper by Bordes et al. for more information.
- Type
torch.Tensor, shape: (n_facts), dtype: torch.int
- evaluated
Indicates if the method LinkPredictionEvaluator.evaluate has already been called.
- Type
bool
- evaluate(b_size, verbose=True)[source]
- Parameters
b_size (int) – Size of the current batch.
verbose (bool) – Indicates whether a progress bar should be displayed during evaluation.
- hit_at_k(k=10)[source]
- Parameters
k (int) – Hit@k is the number of entities that show up in the top k that give facts present in the dataset.
- Returns
avg_hitatk (float) – Average of hit@k for head and tail replacement.
filt_avg_hitatk (float) – Filtered average of hit@k for head and tail replacement.
- mean_rank()[source]
- Returns
mean_rank (float) – Mean rank of the true entity when replacing alternatively head and tail in any fact of the dataset.
filt_mean_rank (float) – Filtered mean rank of the true entity when replacing alternatively head and tail in any fact of the dataset.
Relation Prediction
- class torchkge.evaluation.RelationPredictionEvaluator(model, knowledge_graph, directed=True)[source]
Evaluate performance of given embedding using relation prediction method.
References
Armand Boschin, Thomas Bonald. Enriching Wikidata with Semantified Wikipedia Hyperlinks In proceedings of the Wikidata workshop, ISWC2021, 2021. http://ceur-ws.org/Vol-2982/paper-6.pdf
- Parameters
model (torchkge.models.interfaces.Model) – Embedding model inheriting from the right interface.
knowledge_graph (torchkge.data_structures.KnowledgeGraph) – Knowledge graph on which the evaluation will be done.
directed (bool, optional (default=True)) – Indicates whether the orientation head to tail is known when predicting missing relations. If False, then both tests (h, _, t) and (t, _, r) are done to find the best scoring triples.
- model
Embedding model inheriting from the right interface.
- kg
Knowledge graph on which the evaluation will be done.
- rank_true_rels
For each fact, this is the rank of the true relation when all relations are ranked. They are ranked in decreasing order of scoring function \(f_r(h,t)\).
- Type
torch.Tensor, shape: (n_facts), dtype: torch.int
- filt_rank_true_rels
This is the same as the rank_true_rels when in the filtered case. See referenced paper by Bordes et al. for more information.
- Type
torch.Tensor, shape: (n_facts), dtype: torch.int
- evaluated
Indicates if the method LinkPredictionEvaluator.evaluate has already been called.
- Type
bool
- directed
Indicates whether the orientation head to tail is known when predicting missing relations. If False, then both tests (h, _, t) and (t, _, r) are done to find the best scoring triples.
- Type
bool, optional (default=True)
Triplet Classification
- class torchkge.evaluation.TripletClassificationEvaluator(model, kg_val, kg_test)[source]
Evaluate performance of given embedding using triplet classification method.
References
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning With Neural Tensor Networks for Knowledge Base Completion. In Advances in Neural Information Processing Systems 26, pages 926–934. 2013. https://nlp.stanford.edu/pubs/SocherChenManningNg_NIPS2013.pdf
- Parameters
model (torchkge.models.interfaces.Model) – Embedding model inheriting from the right interface.
kg_val (torchkge.data_structures.KnowledgeGraph) – Knowledge graph on which the validation thresholds will be computed.
kg_test (torchkge.data_structures.KnowledgeGraph) – Knowledge graph on which the testing evaluation will be done.
- model
Embedding model inheriting from the right interface.
- kg_val
Knowledge graph on which the validation thresholds will be computed.
- kg_test
Knowledge graph on which the evaluation will be done.
- evaluated
Indicate whether the evaluate function has been called.
- Type
bool
- thresholds
Value of the thresholds for the scoring function to consider a triplet as true. It is defined by calling the evaluate method.
- Type
float
- sampler
Negative sampler.
- Type
torchkge.sampling.NegativeSampler
- accuracy(b_size)[source]
- Parameters
b_size (int) – Batch size.
- Returns
acc – Share of all triplets (true and negatively sampled ones) that where correctly classified using the thresholds learned from the validation set.
- Return type
float
- evaluate(b_size)[source]
Find relation thresholds using the validation set. As described in the paper by Socher et al., for a relation, the threshold is a value t such that if the score of a triplet is larger than t, the fact is true. If a relation is not present in any fact of the validation set, then the largest value score of all negative samples is used as threshold.
- Parameters
b_size (int) – Batch size.
- get_scores(heads, tails, relations, batch_size)[source]
With head, tail and relation indexes, compute the value of the scoring function of the model.
- Parameters
heads (torch.Tensor, dtype: torch.long, shape: n_facts) – List of heads indices.
tails (torch.Tensor, dtype: torch.long, shape: n_facts) – List of tails indices.
relations (torch.Tensor, dtype: torch.long, shape: n_facts) – List of relation indices.
batch_size (int) –
- Returns
scores – List of scores of each triplet.
- Return type
torch.Tensor, dtype: torch.float, shape: n_facts