javier/Photovoltaic_Fault_Detector
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
Photovoltaic_Fault_Detector/ssd_keras-master/ssd300_evaluation.ipynb

591 lines
334 KiB
Plaintext
Raw Permalink Normal View History

Summary
2020-02-06 16:47:03 -03:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# SSD Evaluation Tutorial\n",
"\n",
"This is a brief tutorial that explains how compute the average precisions for any trained SSD model using the `Evaluator` class. The `Evaluator` computes the average precisions according to the Pascal VOC pre-2010 or post-2010 detection evaluation algorithms. You can find details about these computation methods [here](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/devkit_doc.html#sec:ap).\n",
"\n",
"As an example we'll evaluate an SSD300 on the Pascal VOC 2007 `test` dataset, but note that the `Evaluator` works for any SSD model and any dataset that is compatible with the `DataGenerator`. If you would like to run the evaluation on a different model and/or dataset, the procedure is analogous to what is shown below, you just have to build the appropriate model and load the relevant dataset.\n",
"\n",
"Note: I that in case you would like to evaluate a model on MS COCO, I would recommend to follow the [MS COCO evaluation notebook](https://github.com/pierluigiferrari/ssd_keras/blob/master/ssd300_evaluation_COCO.ipynb) instead, because it can produce the results format required by the MS COCO evaluation server and uses the official MS COCO evaluation code, which computes the mAP slightly differently from the Pascal VOC method.\n",
"\n",
"Note: In case you want to evaluate any of the provided trained models, make sure that you build the respective model with the correct set of scaling factors to reproduce the official results. The models that were trained on MS COCO and fine-tuned on Pascal VOC require the MS COCO scaling factors, not the Pascal VOC scaling factors."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from keras import backend as K\n",
"from keras.models import load_model\n",
"from keras.optimizers import Adam\n",
"from scipy.misc import imread\n",
"import numpy as np\n",
"from matplotlib import pyplot as plt\n",
"\n",
"from models.keras_ssd300 import ssd_300\n",
"from keras_loss_function.keras_ssd_loss import SSDLoss\n",
"from keras_layers.keras_layer_AnchorBoxes import AnchorBoxes\n",
"from keras_layers.keras_layer_DecodeDetections import DecodeDetections\n",
"from keras_layers.keras_layer_DecodeDetectionsFast import DecodeDetectionsFast\n",
"from keras_layers.keras_layer_L2Normalization import L2Normalization\n",
"from data_generator.object_detection_2d_data_generator import DataGenerator\n",
"from eval_utils.average_precision_evaluator import Evaluator\n",
"\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Set a few configuration parameters.\n",
"img_height = 300\n",
"img_width = 300\n",
"n_classes = 20\n",
"model_mode = 'inference'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Load a trained SSD\n",
"\n",
"Either load a trained model or build a model and load trained weights into it. Since the HDF5 files I'm providing contain only the weights for the various SSD versions, not the complete models, you'll have to go with the latter option when using this implementation for the first time. You can then of course save the model and next time load the full model directly, without having to build it.\n",
"\n",
"You can find the download links to all the trained model weights in the README."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.1. Build the model and load trained weights into it"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# 1: Build the Keras model\n",
"\n",
"K.clear_session() # Clear previous models from memory.\n",
"\n",
"model = ssd_300(image_size=(img_height, img_width, 3),\n",
" n_classes=n_classes,\n",
" mode=model_mode,\n",
" l2_regularization=0.0005,\n",
" scales=[0.1, 0.2, 0.37, 0.54, 0.71, 0.88, 1.05], # The scales for MS COCO [0.07, 0.15, 0.33, 0.51, 0.69, 0.87, 1.05]\n",
" aspect_ratios_per_layer=[[1.0, 2.0, 0.5],\n",
" [1.0, 2.0, 0.5, 3.0, 1.0/3.0],\n",
" [1.0, 2.0, 0.5, 3.0, 1.0/3.0],\n",
" [1.0, 2.0, 0.5, 3.0, 1.0/3.0],\n",
" [1.0, 2.0, 0.5],\n",
" [1.0, 2.0, 0.5]],\n",
" two_boxes_for_ar1=True,\n",
" steps=[8, 16, 32, 64, 100, 300],\n",
" offsets=[0.5, 0.5, 0.5, 0.5, 0.5, 0.5],\n",
" clip_boxes=False,\n",
" variances=[0.1, 0.1, 0.2, 0.2],\n",
" normalize_coords=True,\n",
" subtract_mean=[123, 117, 104],\n",
" swap_channels=[2, 1, 0],\n",
" confidence_thresh=0.01,\n",
" iou_threshold=0.45,\n",
" top_k=200,\n",
" nms_max_output_size=400)\n",
"\n",
"# 2: Load the trained weights into the model.\n",
"\n",
"# TODO: Set the path of the trained weights.\n",
"weights_path = 'path/to/trained/weights/VGG_VOC0712_SSD_300x300_ft_iter_120000.h5'\n",
"\n",
"model.load_weights(weights_path, by_name=True)\n",
"\n",
"# 3: Compile the model so that Keras won't complain the next time you load it.\n",
"\n",
"adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)\n",
"\n",
"ssd_loss = SSDLoss(neg_pos_ratio=3, alpha=1.0)\n",
"\n",
"model.compile(optimizer=adam, loss=ssd_loss.compute_loss)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2. Load a trained model\n",
"\n",
"We set `model_mode` to 'inference' above, so the evaluator expects that you load a model that was built in 'inference' mode. If you're loading a model that was built in 'training' mode, change the `model_mode` parameter accordingly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# TODO: Set the path to the `.h5` file of the model to be loaded.\n",
"model_path = 'path/to/trained/model.h5'\n",
"\n",
"# We need to create an SSDLoss object in order to pass that to the model loader.\n",
"ssd_loss = SSDLoss(neg_pos_ratio=3, alpha=1.0)\n",
"\n",
"K.clear_session() # Clear previous models from memory.\n",
"\n",
"model = load_model(model_path, custom_objects={'AnchorBoxes': AnchorBoxes,\n",
" 'L2Normalization': L2Normalization,\n",
" 'DecodeDetections': DecodeDetections,\n",
" 'compute_loss': ssd_loss.compute_loss})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Create a data generator for the evaluation dataset\n",
"\n",
"Instantiate a `DataGenerator` that will serve the evaluation dataset during the prediction phase."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"test.txt: 100%|██████████| 4952/4952 [00:13<00:00, 373.84it/s]\n"
]
}
],
"source": [
"dataset = DataGenerator()\n",
"\n",
"# TODO: Set the paths to the dataset here.\n",
"Pascal_VOC_dataset_images_dir = '../../datasets/VOCdevkit/VOC2007/JPEGImages/'\n",
"Pascal_VOC_dataset_annotations_dir = '../../datasets/VOCdevkit/VOC2007/Annotations/'\n",
"Pascal_VOC_dataset_image_set_filename = '../../datasets/VOCdevkit/VOC2007/ImageSets/Main/test.txt'\n",
"\n",
"# The XML parser needs to now what object class names to look for and in which order to map them to integers.\n",
"classes = ['background',\n",
" 'aeroplane', 'bicycle', 'bird', 'boat',\n",
" 'bottle', 'bus', 'car', 'cat',\n",
" 'chair', 'cow', 'diningtable', 'dog',\n",
" 'horse', 'motorbike', 'person', 'pottedplant',\n",
" 'sheep', 'sofa', 'train', 'tvmonitor']\n",
"\n",
"dataset.parse_xml(images_dirs=[Pascal_VOC_dataset_images_dir],\n",
" image_set_filenames=[Pascal_VOC_dataset_image_set_filename],\n",
" annotations_dirs=[Pascal_VOC_dataset_annotations_dir],\n",
" classes=classes,\n",
" include_classes='all',\n",
" exclude_truncated=False,\n",
" exclude_difficult=False,\n",
" ret=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Run the evaluation\n",
"\n",
"Now that we have instantiated a model and a data generator to serve the dataset, we can set up the evaluator and run the evaluation.\n",
"\n",
"The evaluator is quite flexible: It can compute the average precisions according to the Pascal VOC pre-2010 algorithm, which samples 11 equidistant points of the precision-recall curves, or according to the Pascal VOC post-2010 algorithm, which integrates numerically over the entire precision-recall curves instead of sampling a few individual points. You could also change the number of sampled recall points or the required IoU overlap for a prediction to be considered a true positive, among other things. Check out the `Evaluator`'s documentation for details on all the arguments.\n",
"\n",
"In its default settings, the evaluator's algorithm is identical to the official Pascal VOC pre-2010 Matlab detection evaluation algorithm, so you don't really need to tweak anything unless you want to.\n",
"\n",
"The evaluator roughly performs the following steps: It runs predictions over the entire given dataset, then it matches these predictions to the ground truth boxes, then it computes the precision-recall curves for each class, then it samples 11 equidistant points from these precision-recall curves to compute the average precision for each class, and finally it computes the mean average precision over all classes."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Number of images in the evaluation dataset: 4952\n",
"\n",
"Producing predictions batch-wise: 100%|██████████| 619/619 [02:17<00:00, 4.50it/s]\n",
"Matching predictions to ground truth, class 1/20.: 100%|██████████| 7902/7902 [00:00<00:00, 19253.00it/s]\n",
"Matching predictions to ground truth, class 2/20.: 100%|██████████| 4276/4276 [00:00<00:00, 23249.07it/s]\n",
"Matching predictions to ground truth, class 3/20.: 100%|██████████| 19126/19126 [00:00<00:00, 28311.89it/s]\n",
"Matching predictions to ground truth, class 4/20.: 100%|██████████| 25291/25291 [00:01<00:00, 21126.87it/s]\n",
"Matching predictions to ground truth, class 5/20.: 100%|██████████| 33520/33520 [00:00<00:00, 34410.41it/s]\n",
"Matching predictions to ground truth, class 6/20.: 100%|██████████| 4395/4395 [00:00<00:00, 20824.68it/s]\n",
"Matching predictions to ground truth, class 7/20.: 100%|██████████| 41833/41833 [00:01<00:00, 20956.01it/s]\n",
"Matching predictions to ground truth, class 8/20.: 100%|██████████| 2740/2740 [00:00<00:00, 24270.08it/s]\n",
"Matching predictions to ground truth, class 9/20.: 100%|██████████| 91992/91992 [00:03<00:00, 25723.87it/s]\n",
"Matching predictions to ground truth, class 10/20.: 100%|██████████| 4085/4085 [00:00<00:00, 23969.80it/s]\n",
"Matching predictions to ground truth, class 11/20.: 100%|██████████| 6912/6912 [00:00<00:00, 26573.85it/s]\n",
"Matching predictions to ground truth, class 12/20.: 100%|██████████| 4294/4294 [00:00<00:00, 24942.89it/s]\n",
"Matching predictions to ground truth, class 13/20.: 100%|██████████| 2779/2779 [00:00<00:00, 20814.98it/s]\n",
"Matching predictions to ground truth, class 14/20.: 100%|██████████| 3003/3003 [00:00<00:00, 17807.53it/s]\n",
"Matching predictions to ground truth, class 15/20.: 100%|██████████| 183522/183522 [00:09<00:00, 19243.38it/s]\n",
"Matching predictions to ground truth, class 16/20.: 100%|██████████| 35198/35198 [00:01<00:00, 21565.75it/s]\n",
"Matching predictions to ground truth, class 17/20.: 100%|██████████| 10535/10535 [00:00<00:00, 19680.06it/s]\n",
"Matching predictions to ground truth, class 18/20.: 100%|██████████| 4371/4371 [00:00<00:00, 11523.11it/s]\n",
"Matching predictions to ground truth, class 19/20.: 100%|██████████| 5768/5768 [00:00<00:00, 9747.21it/s]\n",
"Matching predictions to ground truth, class 20/20.: 100%|██████████| 10860/10860 [00:00<00:00, 13970.50it/s]\n",
"Computing precisions and recalls, class 1/20\n",
"Computing precisions and recalls, class 2/20\n",
"Computing precisions and recalls, class 3/20\n",
"Computing precisions and recalls, class 4/20\n",
"Computing precisions and recalls, class 5/20\n",
"Computing precisions and recalls, class 6/20\n",
"Computing precisions and recalls, class 7/20\n",
"Computing precisions and recalls, class 8/20\n",
"Computing precisions and recalls, class 9/20\n",
"Computing precisions and recalls, class 10/20\n",
"Computing precisions and recalls, class 11/20\n",
"Computing precisions and recalls, class 12/20\n",
"Computing precisions and recalls, class 13/20\n",
"Computing precisions and recalls, class 14/20\n",
"Computing precisions and recalls, class 15/20\n",
"Computing precisions and recalls, class 16/20\n",
"Computing precisions and recalls, class 17/20\n",
"Computing precisions and recalls, class 18/20\n",
"Computing precisions and recalls, class 19/20\n",
"Computing precisions and recalls, class 20/20\n",
"Computing average precision, class 1/20\n",
"Computing average precision, class 2/20\n",
"Computing average precision, class 3/20\n",
"Computing average precision, class 4/20\n",
"Computing average precision, class 5/20\n",
"Computing average precision, class 6/20\n",
"Computing average precision, class 7/20\n",
"Computing average precision, class 8/20\n",
"Computing average precision, class 9/20\n",
"Computing average precision, class 10/20\n",
"Computing average precision, class 11/20\n",
"Computing average precision, class 12/20\n",
"Computing average precision, class 13/20\n",
"Computing average precision, class 14/20\n",
"Computing average precision, class 15/20\n",
"Computing average precision, class 16/20\n",
"Computing average precision, class 17/20\n",
"Computing average precision, class 18/20\n",
"Computing average precision, class 19/20\n",
"Computing average precision, class 20/20\n"
]
}
],
"source": [
"evaluator = Evaluator(model=model,\n",
" n_classes=n_classes,\n",
" data_generator=dataset,\n",
" model_mode=model_mode)\n",
"\n",
"results = evaluator(img_height=img_height,\n",
" img_width=img_width,\n",
" batch_size=8,\n",
" data_generator_mode='resize',\n",
" round_confidences=False,\n",
" matching_iou_threshold=0.5,\n",
" border_pixels='include',\n",
" sorting_algorithm='quicksort',\n",
" average_precision_mode='sample',\n",
" num_recall_points=11,\n",
" ignore_neutral_boxes=True,\n",
" return_precisions=True,\n",
" return_recalls=True,\n",
" return_average_precisions=True,\n",
" verbose=True)\n",
"\n",
"mean_average_precision, average_precisions, precisions, recalls = results"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## 4. Visualize the results\n",
"\n",
"Let's take a look:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"aeroplane AP 0.788\n",
"bicycle AP 0.84\n",
"bird AP 0.758\n",
"boat AP 0.693\n",
"bottle AP 0.509\n",
"bus AP 0.868\n",
"car AP 0.858\n",
"cat AP 0.886\n",
"chair AP 0.601\n",
"cow AP 0.822\n",
"diningtable AP 0.764\n",
"dog AP 0.862\n",
"horse AP 0.875\n",
"motorbike AP 0.842\n",
"person AP 0.796\n",
"pottedplant AP 0.526\n",
"sheep AP 0.779\n",
"sofa AP 0.795\n",
"train AP 0.875\n",
"tvmonitor AP 0.773\n",
"\n",
" mAP 0.776\n"
]
}
],
"source": [
"for i in range(1, len(average_precisions)):\n",
" print(\"{:<14}{:<6}{}\".format(classes[i], 'AP', round(average_precisions[i], 3)))\n",
"print()\n",
"print(\"{:<14}{:<6}{}\".format('','mAP', round(mean_average_precision, 3)))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7wAABFACAYAAAD4UuyyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XecXFX9//HXJxtCCCEkARIxoRcxgFJCkCIsKhKQoqhI\nkyIYUUBsP8ECIjbsDRWiIgLyjQ0ENRQFFqJ0BMFQQ6iht0BCICQ5vz/OLJkMu5ud2d25s7Ov5+Nx\nHzNz58yd90xgz3zOvefeSCkhSZIkSVKzGVR0AEmSJEmS+oIFryRJkiSpKVnwSpIkSZKakgWvJEmS\nJKkpWfBKkiRJkpqSBa8kSZIkqSlZ8Ep9JCJSRJxcdI5aRMRBpfy3dPL8uqXn25eFEXFPRPwwIkbV\n+J6DIuILEfFARLwcEf+NiPd343WVWSqX/SvaHxoRN0fECxHxVET8IyLe3sF294yIf0XEc6Xl3xGx\nTy2fTZIGuog4ufQ3efBy2rWW2rXWKVrl+7dFRFud3muH0md9srPvpaI/WxQR90fEbyJifA/e96MR\ncVdEvBIRd0fEUVW8tlt9aFn7FSLi9lL+Izt4fq2I+FNEzC1t8/yIWLvWzyZ1xIJXUkcOLd1uERGb\nd9HuW8B2wK7AWcDHgAsiImp4z68BJwOnAbsD1wF/jIg9lvO6x0oZKpfLgVeAS9sbRsSUUs4bgPcD\nRwJDgH9ExJZl7SYDFwGPAweWlidKn+09NXw2SVL3/If8N/w/RQepg/a+dg1yv9eZs8jfSSvwfWBv\n4PKIWKnaN4yIjwJnAH8GJgN/BH4eER/vxmu71YdW+ByweifbGwZcAWxC/i4+DGwEXBkRK3f/U0ld\n63KUTRoIImLFlNIrRedoFBExDngncDG5Az6U3GF1ZHZK6brS/atKI9RfBbakih8rETGm9B6nppS+\nV1p9ZURsCJwKTO/staV/u+vK15U60UnAX1NKz5U9dRhwXUrp42VtrwCeAfYD2vdoHwLMAT6UUlpc\nancZ8CBwMPD37n42SVL3pZReoOJvejOKiKHkfqeN3F8dCvy1k+Zzyvraf0XEC8BvyX30+VW852Dg\nG8A5KaUvlVZfGRFvBL4WEb9KKb3axSYOo3t9aPtz6wNfBj4K/K6D7X0UWB94U0ppVuk1twH3kgfQ\nf9DdzyZ1xT28amgRsWFEnFM6hGdBRMyOiF90dNhsROwcEZdHxIsRMT8iLo2IzSratJUOU90rIm6J\niFeAT5SeGxERp0XEo2WH+Xy6fG9l2aFW74+Is0qHur4QEb+LiNV647OUtvtIRGwZETMi4qWIuLej\nQ44iYr3Sez9VynxrRLyvyq+50ofJfxu+AvwbOCgiWrr52ptKtxtW+Z67kUeJz61Yfy6weUSsV+X2\n9gVWIf8gKDcEeL5i3UvAqyz793AIMK+92AUo3Z+HfzclqSfeHBFXlvq2xyLilIh47e9qdHJIc0S8\nL/LUknmlfveGiNi79NztEXFB5RuVbWty2bq3RsQFEfFMqS++OyK+0FXgiFgjIk6PiDmlvvau0t7O\nnngvsCrwc+ACYK+Oftt0ota+djvy3uTKvvYcYDVgx+W8vrt9aLtfANOAazrZ3t7kAnpW+4qU0v3k\n3x5OIVKv8YebGt0bgUeBz5IPvTmFvPdxmT1+pcNMLycXJAeTD0FdBZgREWtVbHNj4CfAT8mF1uWl\nzvbvwOHkw4X2Ai4hjy5+o4NcPwIScADwJfIf7T/1xmcpGQGcR+6U9gFuBH4REbuUfea1gOuBtwKf\nLmX4D/Dn9h8BNToUuDOldCNwNvAG4N3dfO36pdvnSxnbf2wctpzXbUo+/HhWxfqZpdsJ3Xz/docC\nT5L/Dcv9HNg1Io6IiJGlvdmnkTvrX5e1mwpsFBFfKv3QWSMiTgLWLbWXJNXmL8A/yQXfecCJwEld\nvSAijiXvyXyS/Pf9g+Qicd1Sk18Ae5b2VJb7GHA/paktETEJuBbYgNxvvofcz3c6HzYiRgD/AvYg\nT7t5D3lP7C9KuWp1KLmvvIjc1w4B9u/yFUtV9rXt57I4eTmv27R0+7+K9d3ta7vbhxIRBwETgeOX\nk6cyS3ueavt9qXMpJReXfrOQD8PfkVxsblm2fhZweUXbEcDTwI/K1rUBS4AtKtruWdrmYRXrf0Uu\nxFYvPW4ttbukot1BpfXvLFuXgJNr+CxnldbtUrZuRfIhQ1PL1v0aeApYrWK7/wBurfH7nVR67y+U\nHo8EFgDTKtqtW2o3pfQ5hpHn8T5GLupXKrXbGVgEHLKc950KPN7B+g1L7/PhKj7DOGAx8INOnv8I\n8HJpu6mUeZsO2u0BPFfW7gVgj6L/H3BxcXHpjwu5WEzACRXrfwm8CIwsPW7vZ1tLj0eUnj+/i22v\nUvobfWLZujVK/fcJZeuuBh4GhnWxrTagrezxiaU+Y6MOcj8NDK7hu1iz1DeeUXo8CHiEvLezsm0i\nD7wPBoYCbwPuBOYDbyy1Wae0vZOW875fLG1vaMX6waX1J3Yj+3L7UGAU+bwXR5Yet/9mOLKi3ULy\nVKbK9/g6sKjo/2ZdmmdxD68aWkQMiYgvlg4fWkAeRZxRevpNpTYbkUdrfxcRg9sX8mE21wI7VWz2\ngZTSrRXrdiIXwudVrD+XPOq6XcX6P1Q8/mPp9ZXtqvosZV5KKV3Z/iDlear3AOVnLpxM3js8t+Jz\nXwq8tTQqXa1DS5/j3NL7Pg9cCOwTEat20P6M0ueYD1xGHniYnFJaUHr9VSmlwSmls2vIUqv2Q7LP\nqnwi8lmWf0bO/S7ynvz/AdPLD3+PiLeRv4Pp5O95MvkIgD+W72WXJFWtsv+cBgwHNuugLcD2peen\ndrbBlNKL5L/ZR5YdHn0YEMCZ8Nq5HXYAfpdSeqmKvJPJR1Pd30Ffuxq17Yk8GGgh79klpdTe724b\nEZW/ByAXqq+SB6CvLd3fI6X0aOn1D5b62lNqyNJt3e1Dge8C91Gx11cqiietUqP7FnAs+fDfa8ij\nvOPJhzYNLbUZU7r9NR3/cX2o4vFjHbQZDTybUlpYsf7xsufLPVH+IKW0MCKeI+9d7Ex3Pku753i9\nVyrajSGfXOmQTt5vNfKId7dERPvhVNcCL0bEyNJTFwAfIp+Q4pcVL/s6uSB+BXgopTS3u+9X4Tlg\nZERESimVrW//3p+tYluHkPdw31a+sjQXeyrwp5TScWXrLwPuIp8lun3+80+BmSmlg8o2cWlE/It8\n+FtnZ6OUJHXtiU4ed9Z/tp8f45HlbPfnwMeBPSLi7+QjkC5IKT1Zen4US/ekVmMM+Wijzk7m1OX5\nOzpxKPm3ycyyvvZC8uG/h5CnSpU7k3zY9iLg4ZTSMzW8Jyz9bTGKZX8LLbev7W4fGhHbkqeHvQNY\nNb+M9gH4lUqfd26pr3+ulKXSaDr+HSTVxIJXjW5/4OyU0tfbV0TE8Io27X/4v0CeF1SpsohNHbR5\nFhgdEUMqit43lD1fbmz5g1KxOIp8Zt/OdOezVOMZ8h7ib3fy/KNVbm8vciezAx13NIfy+oL3wZTS\nTR20rdZM8mHbG7DsPN72kfM7urORiNgGeDN5blalseQfLsvkLQ1W/Lf0unabk39cVLqR/INKklSb\nscDsisfQef/5dOl2HB3P9wQgpfS/iJhBnrf7MrlI/VhZk+fIRzB1NTDdkWfIc4eP6+T5u6vZWERs\nzdK5tB31tR+OiBNLe33bPdaLfS2l9y8veLvT13a3D30zeWChrYNt/KS0jCLPP57J0u+i3ITlZJGq\nYsGrRjeM14+qHl7x+G7gAWDTlNKpNb7PVcD/I58Io/zU+QeRC+ZrK9rvR+kwqZIPkv/AV7Yr153P\nUo1LyIdQz2w/hLiHDiUfmrwPeQ5s5XOHRcQGKaX7euG9Kl1C/m4OIl/WqN3BwP9SPmtjdxxKHgGv\nPDQd8g+LV8gn0XhNabBiC5b9AfZ4ZbuSSXQ9qCFJ6tp+5MvNtduffMLJ2ztpf03p+SmUXVe9Ez8n\nHxo8CrgnpXRF+xMppZd
"text/plain": [
"<matplotlib.figure.Figure at 0x7fc16daea4a8>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"m = max((n_classes + 1) // 2, 2)\n",
"n = 2\n",
"\n",
"fig, cells = plt.subplots(m, n, figsize=(n*8,m*8))\n",
"for i in range(m):\n",
" for j in range(n):\n",
" if n*i+j+1 > n_classes: break\n",
" cells[i, j].plot(recalls[n*i+j+1], precisions[n*i+j+1], color='blue', linewidth=1.0)\n",
" cells[i, j].set_xlabel('recall', fontsize=14)\n",
" cells[i, j].set_ylabel('precision', fontsize=14)\n",
" cells[i, j].grid(True)\n",
" cells[i, j].set_xticks(np.linspace(0,1,11))\n",
" cells[i, j].set_yticks(np.linspace(0,1,11))\n",
" cells[i, j].set_title(\"{}, AP: {:.3f}\".format(classes[n*i+j+1], average_precisions[n*i+j+1]), fontsize=16)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Advanced use\n",
"\n",
"`Evaluator` objects maintain copies of all relevant intermediate results like predictions, precisions and recalls, etc., so in case you want to experiment with different parameters, e.g. different IoU overlaps, there is no need to compute the predictions all over again every time you make a change to a parameter. Instead, you can only update the computation from the point that is affected onwards.\n",
"\n",
"The evaluator's `__call__()` method is just a convenience wrapper that executes its other methods in the correct order. You could just call any of these other methods individually as shown below (but you have to make sure to call them in the correct order).\n",
"\n",
"Note that the example below uses the same evaluator object as above. Say you wanted to compute the Pascal VOC post-2010 'integrate' version of the average precisions instead of the pre-2010 version computed above. The evaluator object still has an internal copy of all the predictions, and since computing the predictions makes up the vast majority of the overall computation time and since the predictions aren't affected by changing the average precision computation mode, we skip computing the predictions again and instead only compute the steps that come after the prediction phase of the evaluation. We could even skip the matching part, since it isn't affected by changing the average precision mode either. In fact, we would only have to call `compute_average_precisions()` `compute_mean_average_precision()` again, but for the sake of illustration we'll re-do the other computations, too."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matching predictions to ground truth, class 1/20.: 100%|██████████| 7902/7902 [00:00<00:00, 19849.68it/s]\n",
"Matching predictions to ground truth, class 2/20.: 100%|██████████| 4276/4276 [00:00<00:00, 21798.36it/s]\n",
"Matching predictions to ground truth, class 3/20.: 100%|██████████| 19126/19126 [00:00<00:00, 28263.72it/s]\n",
"Matching predictions to ground truth, class 4/20.: 100%|██████████| 25291/25291 [00:01<00:00, 20847.78it/s]\n",
"Matching predictions to ground truth, class 5/20.: 100%|██████████| 33520/33520 [00:00<00:00, 34610.95it/s]\n",
"Matching predictions to ground truth, class 6/20.: 100%|██████████| 4395/4395 [00:00<00:00, 23612.98it/s]\n",
"Matching predictions to ground truth, class 7/20.: 100%|██████████| 41833/41833 [00:02<00:00, 20821.01it/s]\n",
"Matching predictions to ground truth, class 8/20.: 100%|██████████| 2740/2740 [00:00<00:00, 25909.74it/s]\n",
"Matching predictions to ground truth, class 9/20.: 100%|██████████| 91992/91992 [00:03<00:00, 25150.58it/s]\n",
"Matching predictions to ground truth, class 10/20.: 100%|██████████| 4085/4085 [00:00<00:00, 22590.90it/s]\n",
"Matching predictions to ground truth, class 11/20.: 100%|██████████| 6912/6912 [00:00<00:00, 28966.61it/s]\n",
"Matching predictions to ground truth, class 12/20.: 100%|██████████| 4294/4294 [00:00<00:00, 23105.94it/s]\n",
"Matching predictions to ground truth, class 13/20.: 100%|██████████| 2779/2779 [00:00<00:00, 20409.40it/s]\n",
"Matching predictions to ground truth, class 14/20.: 100%|██████████| 3003/3003 [00:00<00:00, 17314.28it/s]\n",
"Matching predictions to ground truth, class 15/20.: 100%|██████████| 183522/183522 [00:09<00:00, 18903.68it/s]\n",
"Matching predictions to ground truth, class 16/20.: 100%|██████████| 35198/35198 [00:01<00:00, 26489.65it/s]\n",
"Matching predictions to ground truth, class 17/20.: 100%|██████████| 10535/10535 [00:00<00:00, 28867.54it/s]\n",
"Matching predictions to ground truth, class 18/20.: 100%|██████████| 4371/4371 [00:00<00:00, 22087.65it/s]\n",
"Matching predictions to ground truth, class 19/20.: 100%|██████████| 5768/5768 [00:00<00:00, 17063.02it/s]\n",
"Matching predictions to ground truth, class 20/20.: 100%|██████████| 10860/10860 [00:00<00:00, 25999.09it/s]\n",
"Computing precisions and recalls, class 1/20\n",
"Computing precisions and recalls, class 2/20\n",
"Computing precisions and recalls, class 3/20\n",
"Computing precisions and recalls, class 4/20\n",
"Computing precisions and recalls, class 5/20\n",
"Computing precisions and recalls, class 6/20\n",
"Computing precisions and recalls, class 7/20\n",
"Computing precisions and recalls, class 8/20\n",
"Computing precisions and recalls, class 9/20\n",
"Computing precisions and recalls, class 10/20\n",
"Computing precisions and recalls, class 11/20\n",
"Computing precisions and recalls, class 12/20\n",
"Computing precisions and recalls, class 13/20\n",
"Computing precisions and recalls, class 14/20\n",
"Computing precisions and recalls, class 15/20\n",
"Computing precisions and recalls, class 16/20\n",
"Computing precisions and recalls, class 17/20\n",
"Computing precisions and recalls, class 18/20\n",
"Computing precisions and recalls, class 19/20\n",
"Computing precisions and recalls, class 20/20\n",
"Computing average precision, class 1/20\n",
"Computing average precision, class 2/20\n",
"Computing average precision, class 3/20\n",
"Computing average precision, class 4/20\n",
"Computing average precision, class 5/20\n",
"Computing average precision, class 6/20\n",
"Computing average precision, class 7/20\n",
"Computing average precision, class 8/20\n",
"Computing average precision, class 9/20\n",
"Computing average precision, class 10/20\n",
"Computing average precision, class 11/20\n",
"Computing average precision, class 12/20\n",
"Computing average precision, class 13/20\n",
"Computing average precision, class 14/20\n",
"Computing average precision, class 15/20\n",
"Computing average precision, class 16/20\n",
"Computing average precision, class 17/20\n",
"Computing average precision, class 18/20\n",
"Computing average precision, class 19/20\n",
"Computing average precision, class 20/20\n"
]
}
],
"source": [
"evaluator.get_num_gt_per_class(ignore_neutral_boxes=True,\n",
" verbose=False,\n",
" ret=False)\n",
"\n",
"evaluator.match_predictions(ignore_neutral_boxes=True,\n",
" matching_iou_threshold=0.5,\n",
" border_pixels='include',\n",
" sorting_algorithm='quicksort',\n",
" verbose=True,\n",
" ret=False)\n",
"\n",
"precisions, recalls = evaluator.compute_precision_recall(verbose=True, ret=True)\n",
"\n",
"average_precisions = evaluator.compute_average_precisions(mode='integrate',\n",
" num_recall_points=11,\n",
" verbose=True,\n",
" ret=True)\n",
"\n",
"mean_average_precision = evaluator.compute_mean_average_precision(ret=True)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"aeroplane AP 0.822\n",
"bicycle AP 0.874\n",
"bird AP 0.787\n",
"boat AP 0.713\n",
"bottle AP 0.505\n",
"bus AP 0.899\n",
"car AP 0.89\n",
"cat AP 0.923\n",
"chair AP 0.61\n",
"cow AP 0.845\n",
"diningtable AP 0.79\n",
"dog AP 0.899\n",
"horse AP 0.903\n",
"motorbike AP 0.875\n",
"person AP 0.825\n",
"pottedplant AP 0.526\n",
"sheep AP 0.811\n",
"sofa AP 0.83\n",
"train AP 0.906\n",
"tvmonitor AP 0.797\n",
"\n",
" mAP 0.802\n"
]
}
],
"source": [
"for i in range(1, len(average_precisions)):\n",
" print(\"{:<14}{:<6}{}\".format(classes[i], 'AP', round(average_precisions[i], 3)))\n",
"print()\n",
"print(\"{:<14}{:<6}{}\".format('','mAP', round(mean_average_precision, 3)))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue Copy Permalink