FAQ

Table of Contents

• Training or Test errors
• SVL - training
  • Long training times
  • Under and Over fitting
• Test
  • Improve separation between TP and TN
  • Misclassification errors
  • High number of FALSE NEGATIVE samples
  • High number of FALSE POSITIVE occurrences
  • Long elaboration time
  • A low IoU value
• Different results in SVL and Test
• Not possible to use GPU for training and detection

Training or Test errors

• Labeling verification

  • Verify that no evident and gross labelling errors have been done.

    • Look at the FALSE NEGATIVE samples to verify that you have not set the wrong model of a sample, or that you have added a sample in the wrong position.
    • Look at the FALSE POSITIVE and TRUE NEGATIVE samples to check if you forgot to add a sample.

    If something wrong is founded, correct them, and run the SVL again.
    In some cases, a reset may be necessary. To reduce your training time, in Retina or Surface, you can leave only the models with the wrong samples enabled to reset only them.

  • Inspect the weights of the samples that are not correctly classified. Evaluate if those samples are "border line" and eventually mark them as Optional.

    See also

    sb_t_sample_classify_mode

• Change of the feature set (Retina, Surface)
  • Try to change the Optimization Mode and set it to Medium or Slow. In this case the algorithm itself will execute all the procedures automatically and the feature set will be enlarged.
  • Check if the answers you gave in the feature choice menu have to be changed.
  • If the problem is not already solved, switch the Optimization Mode in Selected (for expert users). In this case, the user is required to have a good knowledge of the descriptors extracted by the available features and he can manually select the set of features that best fit to the model to be trained.

    See also

    Features

  • If training of only a small subset of models does not converge, try disabling all models with positive goodness and run an SVL only on critical models. Before to run the SVL it is necessary to reset it, call sb_svl_reset to reset SVL. The classifier will choose the best recipe that best suits only those specific models.
• Change the network (Deep Cortex, Deep Surface)
  Try to change the current deep learning network with a more complex one.
• Number of epochs (Deep Cortex, Deep Surface)
  Try to increase the number of epochs in order to have a more exhaustive training. This may improve both SVL metrics and test results. In the case of long trainings, try also to enable save best epoch option.
• Scale Levels (Surface)
  With Surface projects try to modify the scale levels configuration.
• Model Size (Retina)
  With Retina projects check if the model has too low resolution for the classifier to recognize the object, or, conversely, if the model has too large resolution for which the classifier, having too much information, is confused and cannot recognize the object. See also sb_t_par_model::obj_size .
• Tile factor / Scale (Deep Surface)
  In Deep Surface projects try to modify the tile factor (if auto tiling is disabled) or scale (if auto tiling is enabled). They directly affect the minimum defect size and defect granularity managed by the SVL and detection. Some simple lineguides are:
  • in case of FALSE NEGATIVE instances on small defects or, more generally, poor defect granularity, try to increase the tile factor or decrease the scale.
  • in case of small FALSE POSITIVE instances, try to reduce the value the tile factor or increase the scale.

    Attention

    Tile factor values > (1, 1) increase training and detection time.

    Scale values equal to (1.0, 1.0) maximize training and detection time.

• Features multiplier (Deep Cortex, Deep Surface)
  Try to increase features multiplier parameter. It increases the algorithm complexity going to augment the number of features used to solve the vision task.
• Too much variability
  If the object has too great variability, it could happen that the classifier cannot model it correctly. However, if the object is presented in n quite well defined positions then n models could be created instead of a single model. For example, an object that always appears straight and upside down, or occurs in n intervals of angle not superimposed on each other. In some situations it may also be better to set the models as collaborative.

SVL - training

For more information about the SVL see section SVL - training.

Long training times

• Thread number
  The number of Thread is a parameter which can considerably improve the training times.

  See also

  Parallel computing

• Device (Deep Cortex, Deep Surface)
  Set type of SVL device to GPU, rather than CPU, may drastically decrease training time. If a GPU device is already used, install a more performing GPU on the machine.

  See also

  Device management

• Decrease ROI area (Retina, Surface)
  • RETINA: if the training dataset contains many images, we suggest training the classifier with a "clever" incremental approach. If the user wants to improve the training quality (e.g. there are errors on Test images), but the number of used training images is considerable, it is advisable to enlarge the training set with images with a reduced ROI including only areas of the image with errors (FALSE POSITIVE, FALSE NEGATIVE).
  • SURFACE: if the images background has low variability, it is advisable to start to reduce ROI area from the beginning. Thus, include in the training set only a limited number of images (2/3) with the ROI extended to the whole surface to be analyzed. In this way, the training is faster because the classifier does not take time to process the section of the image without useful information content.
• Decrease the number of training images
  
  • In Retina or Surface project it is advisable to remove only images in the training set that are not actually used for training. Alternatively move them in the Test set.
• Decrease Goodness Target (Retina, Surface)
  Reduce the value of the Goodness Target parameter to an acceptable value. For such training, especially in those where the detection task requires high generalization capability, high goodness values may be not reachable and may cause overfitting on training data with a degradation of the performance. Obviously, it is important to find an acceptable trade-off: a too low goodness value can lead to underfitting.
• Decrease the number of epochs (Deep Cortex, Deep Surface)
  Training time is directly proportional to the number of epochs. The amount of epochs necessary to reach the best SVL results depends on project complexity.
• Decrease Tile Factor / increase the Scale (Deep Surface)
  Tile factor parameter extends number of images internally processed by SVL by a factor tile_factor.width * tile_factor.height.
  Similarly, when auto tiling is enabled, the closer to (1.0, 1.0) the scale is, the higher will be the number of images internally processed by SVL, especially in case of high resolution images.
  These cases obviously lead to a longer training time.

Under and Over fitting

When we apply Machine Learning method to image vision tasks it is not difficult to perfectly fit our training dataset, especially when we give our algorithm many internal parameters (i.e. degrees of freedom) to fit the observations in the training set. But sometimes, when we evaluate such a complex model on new data, it performs very poorly. In other words, the model does not generalize well. This is a phenomenon called overfitting, and it is one of the biggest challenges of Artificial Intelligence. The problem can be summerized in three possible scenarios, which are shown in the following image. On the left the algorithm has too few parameters and it cannot separate the two classes, on the right the model has too many parameters and it overfalls the training data, in the central case the model has a well-balanced complexity and is able to generalize well.

Under and Over fitting

• Overfitting
  In machine learning, overfitting is "the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably".
  In general to reduce overfitting, try to:
  • add occurrences of the object so that the training dataset is more representative of the variability of the object itself
  In Retina try also to:
  • reduce the size of the model
  • reduce Goodness Target parameter to an acceptable value
  • try to run the SVL with higher search steps, coarse and/or fine, but in detection use lower steps.
• Underfitting
  Occurs when a classifier is unable to adequately acquire the underlying structure of the model with the consequent low image segmentation capacity.
  In general to reduce underfitting, try to:
  • evaluate if same samples are "border line" and eventually mark them as Optional
  In Retina try also to:
  • increase the size of the model
  • increase Goodness Target parameter to an acceptable value

Test

In order to easily test a training, it is necessary to have a correctly labeled test dataset. In this case the sb_project_detection function will be able to classify the results of the analysis and therefore, counters of both the samples found (TRUE POSITIVE and TRUE NEGATIVE) and the detection errors (FALSE NEGATIVE and FALSE POSITIVE) will be updated. See the image below.

Classification counters

If the images are not labeled, all occurrences will be FALSE POSITIVE and TRUE NEGATIVE., the latter only if sb_t_par_model::num_occurrences parameter is different from 0 (see the image below).

Not labeled image with an FALSE POSITIVE and a TRUE_NEGATIVE sample

Having a properly labeled test dataset allows to check very easily if a change in training is improving or worsening. In fact, if the dataset is labeled, counters provide this information, otherwise it is necessary to check all the images one by one, an operation that could take a long time.
To facilitate the labeling operation, which can be very time-consuming, the SB GUI provides a Labeling Assistant tool. For more details see section Labeling assistant with SB GUI.

Improve separation between TP and TN

Attention

Retina projects only

Sometimes, even no detection errors occur on test dataset, it is advisable to reinforce the training going to maximize the weight separation between TRUE POSITIVE and TRUE NEGATIVE instances. In order to have available the information about TRUE NEGATIVE samples in the test images it is necessary to set the sb_t_par_model::num_occurrences to a value different from 0.
A first evaluation can be done observing in the SB_GUI the samples weight chart in the section "Test" (see the image below). It graphically shows the weights distribution of the various metric counter in the interval [-1, 1]. The greater the distance between the maximum TRUE NEGATIVE sample and the minimum TRUE POSITIVE sample the more roboust will be the classifier to distinguish positive and negative instances. In case of low distance or when an extreme of the interval is near to 0 is strongly recommended to move the images containing such samples in the training set and to run an incremental SVL.
To facilitate the user to find the samples corresponding to the outmost weights of the interval, it is possible to filter the images by the weight using the special slider in the section Filters.
The operation have to be repetaed till the user reaches an acceptable separation (i.e. in most of applications a weights separation > 0.4 with the minimum TRUE POSITIVE and the maximum TRUE NEGATIVE respectively higher than 0.1 and lower than -0.1 is advisable).

Weights evaluation of test samples

Misclassification errors

• Background Foreground misunderstanding (Retina)
  If the background has a low degree of variability in the dataset, may be happen that the classifier learns to describe the model depending by the background information and not by the object one. It is possible to check this situation using the weight map of the sample.
  An example of the behaviour may happen when we want to recognize circular objects (like a ring) where the most part of the sample is occupied by the background. In this case, we suggest varying the acquisition background of the training set in order to help the classifier to specialize the training on the ares of the object.
  

  See also

  sb_t_sample_weights_image

• Collaborations
  Check if there are models that can be set as collaborative or, conversely, if models, which are not collaborative, have been set up as if they were.

High number of FALSE NEGATIVE samples

• Check all the conditions listed in the section Training or Test errors.
• Incremental SVL
  To make the training more robust select among FALSE NEGATIVE those samples with lower weight and run an incremental SVL.
• Increase the sensitivity
  to make the classifier more sensitive may cause a decrease of FALSE NEGATIVE numbers, but also an increase of FALSE POSITIVE.

  See also

  sb_project_set_sensitivity

• Collaborations
  in multi-model projects some couples FALSE NEGATIVE - FALSE POSITIVE may be due to occurrences which are actually detected but assigned to wrong model. In this case, take into account if models may be set as collaborative. Indeed, it is important to state if you want to allow a sample can be associated to more than one model (e.g. for an OCR application, uppercase and lowercase letters requires a different labelling because they have different topology. However, the misclassification between the uppercase and lowercase version of the same letter may be not considered as an error). If this is the case, we suggest running the SVL with an appropriate configuration of the collaborations between the models.

  See also

  Collaborations

• Change other parameters
  Retina
  • Speed Boost
    Reduce the value of Speed Boost.
  • Sample scale
    Verify that FALSE NEGATIVE occurrences does not have a scale factor which is not included between the minimum and the maximum ones of the training samples. Otherwise add samples with an appropriate scale to the training set and then run SVL.
  • Perturbations
    Add some perturbation to the model with FALSE NEGATIVE samples and then run SVL.

    See also

    Shallow Learning Perturbations

  • Object minimum distance
    Decrease object minimum distance. Some FALSE NEGATIVE can be caused by samples too close to each other which are filtered because they have a distance less than the minimum set value. However, sb_project_detection may identify such situation and set to 1 the variable sb_t_sample::is_near of FALSE NEGATIVE sample.
  • Sliding window stride
    
    • Reduce Coarse Step and / or Fine Step parameters. These two parameters directly affect the search step of the sliding window. Lower search step parameters may reduce the FALSE NEGATIVE especially those related to model with small dimensions. On the other hand, they increase the elaboration time.

      See also

      How SB works

    • run the SVL with higher search steps, coarse and / or fine, but in detection use lower steps. This solution may be reduced the number of FALSE NEGATIVE due to overfitting.
  Surface, Deep Surface
  • Blobs parameters
    Reduce the minimum of blobs area and/or increase blobs merge distance parameters in sb_t_par.blob_par structure.

High number of FALSE POSITIVE occurrences

• Check all the conditions listed in the section Training or Test errors.
• Incremental SVL
  To make the training more robust select among FALSE POSITIVE those samples with higher weight and run an incremental SVL.
• Reduce the sensitivity
  Make the classifier less sensitive may cause a decrease of FALSE POSITIVE numbers, but also an increase of FALSE NEGATIVE.

  See also

  sb_project_set_sensitivity

• Blobs parameters (Surface, Deep Surface)
  Increase the minimum of blobs area and/or decrease blobs merge distance parameters in sb_t_par.blob_par structure.

Long elaboration time

• Thread number
  The number of Thread is a parameter which can considerably improve analysis times.

  See also

  Parallel computing

• Resize Mode
  Set Resize Mode to SB_INTERPOLATION_MODE_NN instead of SB_INTERPOLATION_MODE_BILINEAR.
• Device (Deep Cortex, Deep Surface)
  Set type of device to GPU, rather than CPU, may drastically decrease inference time. If a GPU device is already used, install a more performing GPU on the machine.

  See also

  Device management

• Decrease ROI area (Retina, Surface)

  • If the object/defect is always located at fixed positions in the image, you can statically define a sub-section of the image on which the classifier will perform the analysis.
  • If the position of the object/defect in the image has a high spatial variability (and we cannot use a static ROI), we suggest using a simple Retina project as object finder (i.e. only for detecting the area of interest). At this point we use the output of the finder to set the ROI and run the "real" analysis only on this portion of the image. Even this solution includes 2 elaborations, it often produces an increase of the testing times performances because finder projects use smaller models, a reduced set of features and smaller search steps and thus have a negligible impact on the overall analysis time.

  See also

  ROI management

• Feature selection (Retina, Surface)
  By playing with the features, it is possible to reduce analysis times while trying not to reduce accuracy.
  • If it is possible try to reduce the number of features used by the classifier if this does not affect its detection capability.
  • Select only those features that better fit to the current detection task or that have faster elaboration time.
  • Reduce the number of different features and, in the best case, use the same features for all models.

    See also

    Features

• Image format (Retina, Surface)
  Work on BW image (if is possible) in such a way that sb_project_detection processes a lower number of channels. Pay attention that the images actually are 8 bit per pixel format.
• Levels (Surface)
  
  • Reduce the number of used scale levels.
  • Use levels with greater scale factor.

    See also

    Levels

• Change other parameters
  Retina
  • Increase the Speed Boost
    The Speed Boost is a parameter which can considerably improve analysis times of the classifier. However, it may cause more FALSE NEGATIVE samples.
  • Model with same size
    In the event that there are multiple models and these have similar dimensions, consider whether it is possible to set the same size for all models or even for groups of models, which is a good compromise between the largest and smallest models.
  • Decrease model size
    Evaluate if it is possible to decrease the model dimensions and use higher scale values.

    See also

    sb_t_par_model.obj_size

    sb_t_sample.scale

  • Reduce the variation of the scale
    If there are occurrences of the model with different sizes, consider whether it is possible to set a model size that allows finding the occurrences by reducing, or eliminating, the scale. As an example we can see the wheels_one_model and wheels_scaler projects.
  • Increase coarse and fine steps
    The greater the coarse and fine step, the shorter the analysis time, but the greater the probability of having FALSE NEGATIVE samples.

    See also

    sb_t_par_model::obj_stride_coarse

    sb_t_par_model::obj_stride_fine

A low IoU value

Attention

Retina projects only

If a TRUE POSITIVE occurrence has a low IoU value this means that the intersection between the sample and the occurrence is small.
In the SB GUI it is possible to set a threshold in order to generate an alarm if the IoU value is below the threshold. You will find the IoU threshold in Setting->Advanced. Usually a value between 50% and 80% is suggested.
If not expressly desired, a low IoU value can be a symptom of possible problems:

• Labeling is not correct or inaccurate
  • Check that no evident and rough labelling errors have been done in the Training set. They may cause inaccurate spatial localization of the test occurrences because the training is corrupted.
  • Check that the ground truth sample associated to the occurrence that generates the warning is correctly located on the object.
  • Verify that all the ground truth samples have a correct scale and, if it is not the case, modify it. Note that, in order the labelling being correct, the object instance must be entirely included in the more internal bounding box of the Sample details window. Errors like this, are probable when the user uses for labelling the Labeling Assistant utility without a proper control on the suggested labelling. See sb_t_par_model.
• Object minimum distance
  Too high a value of object minimum distance can cause the classifier to train itself to find the object inaccurately.

See also

sb_t_sample::IoU

Different results in SVL and Test

Attention

Retina projects only

It may happen that the test performed on a training image does not give the results obtained by the SVL.
In this section we will explain the possible differences and the reasons.
The main cause of the difference is the different handling of the sb_t_par_model::obj_min_distance parameter of the training and in the test functions.

• Training
  The sb_svl_run training was designed to be as sensitive as possible to the sb_t_par_model::obj_min_distance parameter, i.e. minimize by the number of FALSE POSITIVE and FALSE NEGATIVE samples.
  To reduce the number of FALSE POSITIVE samples, the prediction plan is managed as follows: in correspondence with a local maximum of the prediction weight, all other local maxima closest to sb_t_par_model::obj_min_distance but with a lower weight are ignored. Conversely, to reduce the number of FALSE NEGATIVE samples, the sb_t_par_model::obj_min_distance parameter is ignored so as not to filter the samples based on their distance.
• Detection
  The sb_project_detection function uses the sb_t_par_model::obj_min_distance parameter to blindly filter the voting plan so it cannot manage any FALSE POSITIVE samples and FALSE NEGATIVE samples differently. Consequently the higher the parameter, the lower the number of samples FALSE POSITIVE and the greater the number of samples FALSE NEGATIVE.

With these considerations it is possible to understand that the SVL can end without errors but in detection there may be some FALSE NEGATIVE samples. It can happen when there are samples just more distant than the minimum distance but the relative occurrences are instead closer, therefore the one with the lowest weight is eliminated..

Not possible to use GPU for training and detection

Attention

Deep Cortex and Deep Surface projects only.

In this section possible solutions are described to face the problem of having a GPU mounted on the machine but not visible from SB deep learning modules. The problem is not directly notified by the SB library, but the user can easily being aware of it in two ways:

• sb_project_detection or (especially) sb_svl_run are slower than what he expected.
• checking the type of sb_t_svl_res::devices or sb_t_res::devices if corresponds to that set respectively to sb_t_par::devices or sb_t_svl_par.devices .

In order to solve the problem check if installed GPU is supported by the CUDA version used by the SB Deep Learning Framework (releases from 1.10.0 to 1.13.x use CUDA v11.3) or if Nvidia Video driver are correctly installed and its minimum required version satisfied. If it is not the case, update/install the correct driver version.