# Evaluation

## Relation Prediction

class torchkge.evaluation.RelationPredictionEvaluator(model, knowledge_graph, directed=True)[source]

Evaluate performance of given embedding using relation prediction method.

References

Parameters
• model (torchkge.models.interfaces.Model) – Embedding model inheriting from the right interface.

• knowledge_graph (torchkge.data_structures.KnowledgeGraph) – Knowledge graph on which the evaluation will be done.

• directed (bool, optional (default=True)) – Indicates whether the orientation head to tail is known when predicting missing relations. If False, then both tests (h, _, t) and (t, _, r) are done to find the best scoring triples.

model

Embedding model inheriting from the right interface.

Type

torchkge.models.interfaces.Model

kg

Knowledge graph on which the evaluation will be done.

Type

torchkge.data_structures.KnowledgeGraph

rank_true_rels

For each fact, this is the rank of the true relation when all relations are ranked. They are ranked in decreasing order of scoring function $$f_r(h,t)$$.

Type

torch.Tensor, shape: (n_facts), dtype: torch.int

filt_rank_true_rels

This is the same as the rank_true_rels when in the filtered case. See referenced paper by Bordes et al. for more information.

Type

torch.Tensor, shape: (n_facts), dtype: torch.int

evaluated

Type

bool

directed

Indicates whether the orientation head to tail is known when predicting missing relations. If False, then both tests (h, _, t) and (t, _, r) are done to find the best scoring triples.

Type

bool, optional (default=True)

## Triplet Classification

class torchkge.evaluation.TripletClassificationEvaluator(model, kg_val, kg_test)[source]

Evaluate performance of given embedding using triplet classification method.

References

Parameters
model

Embedding model inheriting from the right interface.

Type

torchkge.models.interfaces.Model

kg_val

Knowledge graph on which the validation thresholds will be computed.

Type

torchkge.data_structures.KnowledgeGraph

kg_test

Knowledge graph on which the evaluation will be done.

Type

torchkge.data_structures.KnowledgeGraph

evaluated

Indicate whether the evaluate function has been called.

Type

bool

thresholds

Value of the thresholds for the scoring function to consider a triplet as true. It is defined by calling the evaluate method.

Type

float

sampler

Negative sampler.

Type

torchkge.sampling.NegativeSampler

accuracy(b_size)[source]
Parameters

b_size (int) – Batch size.

Returns

acc – Share of all triplets (true and negatively sampled ones) that where correctly classified using the thresholds learned from the validation set.

Return type

float

evaluate(b_size)[source]

Find relation thresholds using the validation set. As described in the paper by Socher et al., for a relation, the threshold is a value t such that if the score of a triplet is larger than t, the fact is true. If a relation is not present in any fact of the validation set, then the largest value score of all negative samples is used as threshold.

Parameters

b_size (int) – Batch size.

With head, tail and relation indexes, compute the value of the scoring function of the model.

Parameters
• heads (torch.Tensor, dtype: torch.long, shape: n_facts) – List of heads indices.

• tails (torch.Tensor, dtype: torch.long, shape: n_facts) – List of tails indices.

• relations (torch.Tensor, dtype: torch.long, shape: n_facts) – List of relation indices.

• batch_size (int) –

Returns

scores – List of scores of each triplet.

Return type

torch.Tensor, dtype: torch.float, shape: n_facts