imaginaire.evaluation package

Submodules

imaginaire.evaluation.common module

class imaginaire.evaluation.common.CleanInceptionV3[source]

Bases: torch.nn.modules.module.Module

forward(img_batch, transform=True, **_kwargs)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
imaginaire.evaluation.common.clean_resize(img_batch)[source]
imaginaire.evaluation.common.compute_all_metrics(act_dir, data_loader, net_G, key_real='images', key_fake='fake_images', sample_size=None, preprocess=None, is_video=False, few_shot_video=False, kid_num_subsets=1, kid_subset_size=None, key_prefix='', prdc_k=5, metrics=None, dataset_name='', aws_credentials=None, **kwargs)[source]
Parameters

  • act_dir (string) – Path to a directory to temporarily save feature activations.

  • data_loader (obj) – PyTorch dataloader object.

  • net_G (obj) – The generator module.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • sample_size (int or None) – How many samples to use for FID.

  • preprocess (func or None) – Pre-processing function to use.

  • is_video (bool) – Whether we are handling video sequences.

  • few_shot_video (bool) – If True, uses few-shot video synthesis.

  • kid_num_subsets (int) – Number of subsets for KID evaluation.

  • kid_subset_size (int or None) – The number of samples in each subset for KID evaluation.

  • key_prefix (string) – Add this string before all keys of the output dictionary.

  • prdc_k (int) – The K used for computing K-NN when evaluating precision/recall/density/coverage.

  • metrics (list of strings) – Which metrics we want to evaluate.

  • dataset_name (string) – The name of the dataset, currently only used to determine which segmentation network to use for segmentation evaluation.

Returns

Inception features of the current batch. Note that

only the master gpu will get it.

Return type

batch_y (tensor)

imaginaire.evaluation.common.compute_all_metrics_data(data_loader_a, data_loader_b, key_a='images', key_b='images', sample_size=None, preprocess=None, kid_num_subsets=1, kid_subset_size=None, key_prefix='', prdc_k=5, metrics=None, dataset_name='', aws_credentials=None, **kwargs)[source]
Parameters

  • act_dir (string) – Path to a directory to temporarily save feature activations.

  • data_loader (obj) – PyTorch dataloader object.

  • net_G (obj) – The generator module.

  • key_a (str) – Dictionary key value for the real data.

  • key_b (str) – Dictionary key value for the fake data.

  • sample_size (int or None) – How many samples to use for FID.

  • preprocess (func or None) – Pre-processing function to use.

  • is_video (bool) – Whether we are handling video sequences.

  • few_shot_video (bool) – If True, uses few-shot video synthesis.

  • kid_num_subsets (int) – Number of subsets for KID evaluation.

  • kid_subset_size (int or None) – The number of samples in each subset for KID evaluation.

  • key_prefix (string) – Add this string before all keys of the output dictionary.

  • prdc_k (int) – The K used for computing K-NN when evaluating precision/recall/density/coverage.

  • metrics (list of strings) – Which metrics we want to evaluate.

  • dataset_name (string) – The name of the dataset, currently only used to determine which segmentation network to use for segmentation evaluation.

Returns

Inception features of the current batch. Note that

only the master gpu will get it.

Return type

batch_y (tensor)

imaginaire.evaluation.common.compute_nn(input_features, k, num_splits=50)[source]
imaginaire.evaluation.common.compute_pairwise_distance(data_x, data_y=None, num_splits=10)[source]
Parameters

  • data_x – numpy.ndarray([N, feature_dim], dtype=np.float32)

  • data_y – numpy.ndarray([N, feature_dim], dtype=np.float32)

Returns

numpy.ndarray([N, N], dtype=np.float32) of pairwise distances.

imaginaire.evaluation.common.gather_tensors(batch_y)[source]
imaginaire.evaluation.common.get_activations(data_loader, key_real, key_fake, generator=None, sample_size=None, preprocess=None, align_corners=True, network='inception', **kwargs)[source]

Compute activation values and pack them in a list.

Parameters

  • data_loader (obj) – PyTorch dataloader object.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • generator (obj) – PyTorch trainer network.

  • sample_size (int) – How many samples to use for FID.

  • preprocess (func) – Pre-processing function to use.

  • align_corners (bool) – The 'align_corners' parameter to be used for torch.nn.functional.interpolate.

Returns

Inception features of the current batch. Note that

only the master gpu will get it.

Return type

batch_y (tensor)

imaginaire.evaluation.common.get_outputs(data_loader, key_real, key_fake, generator=None, sample_size=None, preprocess=None, align_corners=True, network='inception', output_module_dict=None, **kwargs)[source]

Compute activation values and pack them in a list.

Parameters

  • data_loader (obj) – PyTorch dataloader object.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • generator (obj) – PyTorch trainer network.

  • sample_size (int) – How many samples to use for FID.

  • preprocess (func) – Pre-processing function to use.

  • align_corners (bool) – The 'align_corners' parameter to be used for torch.nn.functional.interpolate.

Returns

Inception features of the current batch. Note that

only the master gpu will get it.

Return type

batch_y (tensor)

imaginaire.evaluation.common.get_video_activations(data_loader, key_real, key_fake, trainer=None, sample_size=None, preprocess=None, few_shot=False)[source]

Compute activation values and pack them in a list. We do not do all reduce here.

Parameters

  • data_loader (obj) – PyTorch dataloader object.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • trainer (obj) – Trainer. Video generation is more involved, we rely on the “reset” and “test” function to conduct the evaluation.

  • sample_size (int) – For computing video activation, we will use .

  • preprocess (func) – The preprocess function to be applied to the data.

  • few_shot (bool) – If True, uses the few-shot setting.

Returns

Inception features of the current batch. Note that

only the master gpu will get it.

Return type

batch_y (tensor)

imaginaire.evaluation.common.inception_forward(inception, images)[source]
imaginaire.evaluation.common.inception_init()[source]
imaginaire.evaluation.common.load_or_compute_activations(act_path, data_loader, key_real, key_fake, generator=None, sample_size=None, preprocess=None, is_video=False, few_shot_video=False, **kwargs)[source]

Load mean and covariance from saved npy file if exists. Otherwise, compute the mean and covariance.

Parameters

  • act_path (str or None) – Location for the numpy file to store or to load the activations.

  • data_loader (obj) – PyTorch dataloader object.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • generator (obj) – PyTorch trainer network.

  • sample_size (int) – How many samples to be used for computing the KID.

  • preprocess (func) – The preprocess function to be applied to the data.

  • is_video (bool) – Whether we are handling video sequences.

  • few_shot_video (bool) – If True, uses few-shot video synthesis.

Returns

(torch.Tensor) Feature activations.

imaginaire.evaluation.common.set_sequence_idx(few_shot, data_loader, sequence_idx)[source]

Get sequence index

Parameters

  • few_shot (bool) – If True, uses the few-shot setting.

  • data_loader – dataloader object

  • sequence_idx (int) – which sequence to use.

imaginaire.evaluation.fid module

imaginaire.evaluation.fid.compute_fid(fid_path, data_loader, net_G, key_real='images', key_fake='fake_images', sample_size=None, preprocess=None, return_act=False, is_video=False, few_shot_video=False, **kwargs)[source]

Compute the fid score.

Parameters

  • fid_path (str) – Location for the numpy file to store or to load the statistics.

  • data_loader (obj) – PyTorch dataloader object.

  • net_G (obj) – For image generation modes, net_G is the generator network. For video generation models, net_G is the trainer.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • sample_size (int or tuple) – How many samples to be used.

  • preprocess (func) – The preprocess function to be applied to the data.

  • return_act (bool) – If True, also returns feature activations of real and fake data.

  • is_video (bool) – Whether we are handling video sequences.

  • few_shot_video (bool) – If True, uses few-shot video synthesis.

Returns

FID value.

Return type

(float)

imaginaire.evaluation.fid.compute_fid_data(fid_path, data_loader_a, data_loader_b, key_a='images', key_b='images', sample_size=None, is_video=False, few_shot_video=False, **kwargs)[source]

Compute the fid score between two datasets.

Parameters

  • fid_path (str) – Location for the numpy file to store or to load the statistics.

  • data_loader_a (obj) – PyTorch dataloader object for dataset a.

  • data_loader_b (obj) – PyTorch dataloader object for dataset b.

  • key_a (str) – Dictionary key value for images in the dataset a.

  • key_b (str) – Dictionary key value for images in the dataset b.

  • sample_size (int) – How many samples to be used for computing the FID.

  • is_video (bool) – Whether we are handling video sequences.

  • few_shot_video (bool) – If True, uses few-shot video synthesis.

Returns

FID value.

Return type

(float)

imaginaire.evaluation.kid module

Modified from https://github.com/abdulfatir/gan-metrics-pytorch Copyright 2018 Institute of Bioinformatics, JKU Linz Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

imaginaire.evaluation.kid.compute_kid(kid_path, data_loader, net_G, key_real='images', key_fake='fake_images', real_act=None, fake_act=None, sample_size=None, preprocess=None, is_video=False, save_act=True, num_subsets=1, subset_size=None, **kwargs)[source]

Compute the kid score.

Parameters

  • kid_path (str) – Location for store feature activations.

  • data_loader (obj) – PyTorch dataloader object.

  • net_G (obj) – For image generation modes, net_G is the PyTorch trainer network. For video generation models, net_G is the trainer because video generation requires more complicated processing.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • real_act (torch.Tensor or None) – Feature activations of real data.

  • fake_act (torch.Tensor or None) – Feature activations of fake data.

  • sample_size (int) – How many samples to be used for computing feature activations.

  • preprocess (func) – The preprocess function to be applied to the data.

  • is_video (bool) – Whether we are handling video sequences.

  • save_act (bool) – If True, saves real activations to the disk and reload them in the future. It might save some computation but will cost storage.

  • num_subsets (int) – Number of subsets to sample from all the samples.

  • subset_size (int) – Number of samples in each subset.

Returns

KID value.

Return type

kid (float)

imaginaire.evaluation.kid.compute_kid_data(kid_path, data_loader_a, data_loader_b, key_a='images', key_b='images', sample_size=None, is_video=False, num_subsets=1, subset_size=None, **kwargs)[source]

Compute the kid score between two datasets.

Parameters

  • kid_path (str) – Location for store feature activations.

  • data_loader_a (obj) – PyTorch dataloader object for dataset a.

  • data_loader_b (obj) – PyTorch dataloader object for dataset b.

  • key_a (str) – Dictionary key value for images in the dataset a.

  • key_b (str) – Dictionary key value for images in the dataset b.

  • sample_size (int) – How many samples to be used for computing the KID.

  • is_video (bool) – Whether we are handling video sequences.

  • num_subsets (int) – Number of subsets to sample from the whole data.

  • subset_size (int) – Number of samples in each subset.

Returns

KID value.

Return type

kid (float)

imaginaire.evaluation.knn module

imaginaire.evaluation.lpips module

class imaginaire.evaluation.lpips.LPIPSNet[source]

Bases: torch.nn.modules.module.Module

forward(fake_images, fake_images_another, align_corners=True)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.lpips.LPNet[source]

Bases: torch.nn.modules.module.Module

forward(in0, avg=False)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.lpips.NetLinLayer(dim)[source]

Bases: torch.nn.modules.module.Module

A single linear layer used as placeholder for LPIPS learnt weights

forward(inp)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.lpips.ScalingLayer[source]

Bases: torch.nn.modules.module.Module

forward(inp)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
imaginaire.evaluation.lpips.get_lpips_model()[source]
imaginaire.evaluation.lpips.normalize_tensor(in_feat, eps=1e-05)[source]
class imaginaire.evaluation.lpips.vgg16(requires_grad=False, pretrained=True)[source]

Bases: torch.nn.modules.module.Module

forward(x)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None

imaginaire.evaluation.msid module

imaginaire.evaluation.msid.construct_graph(input_features, k, num_splits=10)[source]
imaginaire.evaluation.msid.construct_graph_kgraph(data, k)[source]
imaginaire.evaluation.msid.construct_graph_sparse(data, k)[source]
imaginaire.evaluation.msid.msid_descriptor(x, ts=array([ 0.1, 0.10182235, 0.10367792, 0.1055673, 0.10749111, 0.10944998, 0.11144455, 0.11347546, 0.11554339, 0.117649, 0.11979298, 0.12197603, 0.12419887, 0.12646222, 0.12876681, 0.13111339, 0.13350275, 0.13593564, 0.13841287, 0.14093524, 0.14350358, 0.14611873, 0.14878153, 0.15149286, 0.15425359, 0.15706464, 0.15992692, 0.16284135, 0.1658089, 0.16883053, 0.17190722, 0.17503998, 0.17822983, 0.18147781, 0.18478498, 0.18815242, 0.19158122, 0.19507251, 0.19862743, 0.20224712, 0.20593278, 0.20968561, 0.21350683, 0.21739768, 0.22135944, 0.22539339, 0.22950086, 0.23368318, 0.23794172, 0.24227786, 0.24669302, 0.25118864, 0.25576619, 0.26042716, 0.26517307, 0.27000546, 0.27492592, 0.27993605, 0.28503747, 0.29023187, 0.29552092, 0.30090636, 0.30638995, 0.31197346, 0.31765872, 0.32344759, 0.32934195, 0.33534373, 0.34145489, 0.34767741, 0.35401332, 0.3604647, 0.36703365, 0.37372231, 0.38053285, 0.38746751, 0.39452855, 0.40171826, 0.40903899, 0.41649313, 0.42408311, 0.43181141, 0.43968055, 0.44769309, 0.45585165, 0.46415888, 0.47261751, 0.48123027, 0.49, 0.49892954, 0.5080218, 0.51727977, 0.52670644, 0.5363049, 0.54607828, 0.55602976, 0.5661626, 0.57648009, 0.58698561, 0.59768257, 0.60857446, 0.61966485, 0.63095734, 0.64245563, 0.65416345, 0.66608463, 0.67822306, 0.69058269, 0.70316755, 0.71598176, 0.72902949, 0.742315, 0.75584261, 0.76961675, 0.7836419, 0.79792263, 0.81246362, 0.82726959, 0.84234538, 0.8576959, 0.87332616, 0.88924127, 0.9054464, 0.92194684, 0.93874799, 0.95585531, 0.97327439, 0.9910109, 1.00907064, 1.02745949, 1.04618344, 1.06524862, 1.08466123, 1.10442761, 1.1245542, 1.14504757, 1.1659144, 1.1871615, 1.2087958, 1.23082435, 1.25325434, 1.27609308, 1.29934802, 1.32302676, 1.347137, 1.37168662, 1.39668362, 1.42213615, 1.44805252, 1.47444118, 1.50131073, 1.52866994, 1.55652773, 1.58489319, 1.61377557, 1.64318429, 1.67312894, 1.70361929, 1.73466528, 1.76627704, 1.79846488, 1.83123929, 1.86461097, 1.8985908, 1.93318987, 1.96841945, 2.00429104, 2.04081633, 2.07800725, 2.11587592, 2.15443469, 2.19369614, 2.23367307, 2.27437852, 2.31582577, 2.35802834, 2.40099998, 2.44475472, 2.48930683, 2.53467084, 2.58086154, 2.627894, 2.67578356, 2.72454583, 2.77419673, 2.82475244, 2.87622945, 2.92864456, 2.98201486, 3.03635776, 3.09169098, 3.14803256, 3.20540089, 3.26381467, 3.32329296, 3.38385515, 3.44552101, 3.50831063, 3.5722445, 3.63734348, 3.70362879, 3.77112205, 3.83984528, 3.90982089, 3.98107171, 4.05362096, 4.12749233, 4.20270989, 4.27929818, 4.35728218, 4.43668733, 4.51753952, 4.59986513, 4.683691, 4.76904447, 4.85595339, 4.9444461, 5.03455146, 5.12629886, 5.21971822, 5.31484001, 5.41169527, 5.51031556, 5.61073307, 5.71298054, 5.81709133, 5.92309938, 6.03103928, 6.14094622, 6.25285606, 6.36680529, 6.48283108, 6.60097128, 6.72126441, 6.8437497, 6.96846712, 7.09545732, 7.22476174, 7.35642254, 7.49048268, 7.62698586, 7.76597661, 7.90750027, 8.051603, 8.19833179, 8.34773449, 8.49985985, 8.65475746, 8.81247787, 8.97307249, 9.13659373, 9.3030949, 9.47263031, 9.64525526, 9.82102604, 10. ]), k=5, m=10, niters=100, rademacher=False, graph_builder='full', normalized_laplacian=True, normalize='empty')[source]

Compute the msid descriptor for a single sample x :param x: x samples :param ts: temperature values :param k: number of neighbours for graph construction :param m: Lanczos steps in SLQ :param niters: number of starting random vectors for SLQ :param rademacher: if True, sample random vectors from Rademacher

distributions, else sample standard normal distribution

Parameters

  • graph_builder – if ‘kgraph’, uses faster graph construction (options: ‘ rse’, ‘kgraph’)

  • normalized_laplacian – if True, use normalized Laplacian

  • normalize – ‘empty’ for average heat kernel (corresponds to the empty graph normalization of NetLSD), ‘complete’ for the complete, ‘er’ for erdos-renyi normalization, ‘none’ for no normalization

Returns

normalized msid descriptor

Return type

normed_msidx

imaginaire.evaluation.msid.np_euc_cdist(data)[source]
imaginaire.evaluation.msid.slq_red_var(A, m, niters, ts, rademacher)[source]

Compute the trace of matrix exponential with reduced variance

Parameters

  • A – square matrix in trace(exp(-t*A)), where t is temperature

  • m – number of Lanczos steps

  • niters – number of quadratures (also, the number of random vectors in the hutchinson trace estimator)

  • ts – an array with temperatures

Returns

estimate of trace for each temperature value in ts

Return type

traces

imaginaire.evaluation.prdc module

Modified from https://github.com/clovaai/generative-evaluation-prdc Copyright (c) 2020-present NAVER Corp. MIT license

imaginaire.evaluation.prdc.compute_prdc(prdc_path, data_loader, net_G, key_real='images', key_fake='fake_images', real_act=None, fake_act=None, sample_size=None, save_act=True, k=10, **kwargs)[source]

Compute precision diversity curve

Args:

imaginaire.evaluation.prdc.get_kth_value(unsorted, k, dim=-1)[source]
Parameters

  • unsorted – numpy.ndarray of any dimensionality.

  • k – int

Returns

kth values along the designated axis.

imaginaire.evaluation.pretrained module

Modified from https://github.com/mseitzer/pytorch-fid

Code adapted from https://github.com/bioinf-jku/TTUR to use PyTorch instead of Tensorflow Copyright 2018 Institute of Bioinformatics, JKU Linz Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

class imaginaire.evaluation.pretrained.FIDInceptionA(in_channels, pool_features)[source]

Bases: torchvision.models.inception.InceptionA

InceptionA block patched for FID computation

forward(x)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.pretrained.FIDInceptionC(in_channels, channels_7x7)[source]

Bases: torchvision.models.inception.InceptionC

InceptionC block patched for FID computation

forward(x)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.pretrained.FIDInceptionE_1(in_channels)[source]

Bases: torchvision.models.inception.InceptionE

First InceptionE block patched for FID computation

forward(x)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.pretrained.FIDInceptionE_2(in_channels)[source]

Bases: torchvision.models.inception.InceptionE

Second InceptionE block patched for FID computation

forward(x)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.pretrained.InceptionV3[source]

Bases: torch.nn.modules.module.Module

forward(x, align_corners=True)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.pretrained.SwAV[source]

Bases: torch.nn.modules.module.Module

forward(x, align_corners=True)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.pretrained.TFInceptionV3[source]

Bases: torch.nn.modules.module.Module

forward(x, align_corners=True)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None
class imaginaire.evaluation.pretrained.Vgg16[source]

Bases: torch.nn.modules.module.Module

forward(x, align_corners=True)[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training = None

Module contents

imaginaire.evaluation.compute_fid(fid_path, data_loader, net_G, key_real='images', key_fake='fake_images', sample_size=None, preprocess=None, return_act=False, is_video=False, few_shot_video=False, **kwargs)[source]

Compute the fid score.

Parameters

  • fid_path (str) – Location for the numpy file to store or to load the statistics.

  • data_loader (obj) – PyTorch dataloader object.

  • net_G (obj) – For image generation modes, net_G is the generator network. For video generation models, net_G is the trainer.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • sample_size (int or tuple) – How many samples to be used.

  • preprocess (func) – The preprocess function to be applied to the data.

  • return_act (bool) – If True, also returns feature activations of real and fake data.

  • is_video (bool) – Whether we are handling video sequences.

  • few_shot_video (bool) – If True, uses few-shot video synthesis.

Returns

FID value.

Return type

(float)

imaginaire.evaluation.compute_fid_data(fid_path, data_loader_a, data_loader_b, key_a='images', key_b='images', sample_size=None, is_video=False, few_shot_video=False, **kwargs)[source]

Compute the fid score between two datasets.

Parameters

  • fid_path (str) – Location for the numpy file to store or to load the statistics.

  • data_loader_a (obj) – PyTorch dataloader object for dataset a.

  • data_loader_b (obj) – PyTorch dataloader object for dataset b.

  • key_a (str) – Dictionary key value for images in the dataset a.

  • key_b (str) – Dictionary key value for images in the dataset b.

  • sample_size (int) – How many samples to be used for computing the FID.

  • is_video (bool) – Whether we are handling video sequences.

  • few_shot_video (bool) – If True, uses few-shot video synthesis.

Returns

FID value.

Return type

(float)

imaginaire.evaluation.compute_kid(kid_path, data_loader, net_G, key_real='images', key_fake='fake_images', real_act=None, fake_act=None, sample_size=None, preprocess=None, is_video=False, save_act=True, num_subsets=1, subset_size=None, **kwargs)[source]

Compute the kid score.

Parameters

  • kid_path (str) – Location for store feature activations.

  • data_loader (obj) – PyTorch dataloader object.

  • net_G (obj) – For image generation modes, net_G is the PyTorch trainer network. For video generation models, net_G is the trainer because video generation requires more complicated processing.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • real_act (torch.Tensor or None) – Feature activations of real data.

  • fake_act (torch.Tensor or None) – Feature activations of fake data.

  • sample_size (int) – How many samples to be used for computing feature activations.

  • preprocess (func) – The preprocess function to be applied to the data.

  • is_video (bool) – Whether we are handling video sequences.

  • save_act (bool) – If True, saves real activations to the disk and reload them in the future. It might save some computation but will cost storage.

  • num_subsets (int) – Number of subsets to sample from all the samples.

  • subset_size (int) – Number of samples in each subset.

Returns

KID value.

Return type

kid (float)

imaginaire.evaluation.compute_kid_data(kid_path, data_loader_a, data_loader_b, key_a='images', key_b='images', sample_size=None, is_video=False, num_subsets=1, subset_size=None, **kwargs)[source]

Compute the kid score between two datasets.

Parameters

  • kid_path (str) – Location for store feature activations.

  • data_loader_a (obj) – PyTorch dataloader object for dataset a.

  • data_loader_b (obj) – PyTorch dataloader object for dataset b.

  • key_a (str) – Dictionary key value for images in the dataset a.

  • key_b (str) – Dictionary key value for images in the dataset b.

  • sample_size (int) – How many samples to be used for computing the KID.

  • is_video (bool) – Whether we are handling video sequences.

  • num_subsets (int) – Number of subsets to sample from the whole data.

  • subset_size (int) – Number of samples in each subset.

Returns

KID value.

Return type

kid (float)

imaginaire.evaluation.compute_prdc(prdc_path, data_loader, net_G, key_real='images', key_fake='fake_images', real_act=None, fake_act=None, sample_size=None, save_act=True, k=10, **kwargs)[source]

Compute precision diversity curve

Args:

imaginaire.evaluation.compute_all_metrics(act_dir, data_loader, net_G, key_real='images', key_fake='fake_images', sample_size=None, preprocess=None, is_video=False, few_shot_video=False, kid_num_subsets=1, kid_subset_size=None, key_prefix='', prdc_k=5, metrics=None, dataset_name='', aws_credentials=None, **kwargs)[source]
Parameters

  • act_dir (string) – Path to a directory to temporarily save feature activations.

  • data_loader (obj) – PyTorch dataloader object.

  • net_G (obj) – The generator module.

  • key_real (str) – Dictionary key value for the real data.

  • key_fake (str) – Dictionary key value for the fake data.

  • sample_size (int or None) – How many samples to use for FID.

  • preprocess (func or None) – Pre-processing function to use.

  • is_video (bool) – Whether we are handling video sequences.

  • few_shot_video (bool) – If True, uses few-shot video synthesis.

  • kid_num_subsets (int) – Number of subsets for KID evaluation.

  • kid_subset_size (int or None) – The number of samples in each subset for KID evaluation.

  • key_prefix (string) – Add this string before all keys of the output dictionary.

  • prdc_k (int) – The K used for computing K-NN when evaluating precision/recall/density/coverage.

  • metrics (list of strings) – Which metrics we want to evaluate.

  • dataset_name (string) – The name of the dataset, currently only used to determine which segmentation network to use for segmentation evaluation.

Returns

Inception features of the current batch. Note that

only the master gpu will get it.

Return type

batch_y (tensor)

imaginaire.evaluation.compute_all_metrics_data(data_loader_a, data_loader_b, key_a='images', key_b='images', sample_size=None, preprocess=None, kid_num_subsets=1, kid_subset_size=None, key_prefix='', prdc_k=5, metrics=None, dataset_name='', aws_credentials=None, **kwargs)[source]
Parameters

  • act_dir (string) – Path to a directory to temporarily save feature activations.

  • data_loader (obj) – PyTorch dataloader object.

  • net_G (obj) – The generator module.

  • key_a (str) – Dictionary key value for the real data.

  • key_b (str) – Dictionary key value for the fake data.

  • sample_size (int or None) – How many samples to use for FID.

  • preprocess (func or None) – Pre-processing function to use.

  • is_video (bool) – Whether we are handling video sequences.

  • few_shot_video (bool) – If True, uses few-shot video synthesis.

  • kid_num_subsets (int) – Number of subsets for KID evaluation.

  • kid_subset_size (int or None) – The number of samples in each subset for KID evaluation.

  • key_prefix (string) – Add this string before all keys of the output dictionary.

  • prdc_k (int) – The K used for computing K-NN when evaluating precision/recall/density/coverage.

  • metrics (list of strings) – Which metrics we want to evaluate.

  • dataset_name (string) – The name of the dataset, currently only used to determine which segmentation network to use for segmentation evaluation.

Returns

Inception features of the current batch. Note that

only the master gpu will get it.

Return type

batch_y (tensor)