mmdet.apis

class mmdet.apis.DetInferencer(model: Optional[Union[dict, Config, ConfigDict, str]] = None, weights: Optional[str] = None, device: Optional[str] = None, scope: Optional[str] = 'mmdet', palette: str = 'none', show_progress: bool = True)

Object Detection Inferencer.

Parameters

• model (str, optional) – Path to the config file or the model name defined in metafile. For example, it could be “rtmdet-s” or ‘rtmdet_s_8xb32-300e_coco’ or “configs/rtmdet/rtmdet_s_8xb32-300e_coco.py”. If model is not specified, user must provide the weights saved by MMEngine which contains the config string. Defaults to None.

• weights (str, optional) – Path to the checkpoint. If it is not specified and model is a model name of metafile, the weights will be loaded from metafile. Defaults to None.

• device (str, optional) – Device to run inference. If None, the available device will be automatically used. Defaults to None.

• scope (str, optional) – The scope of the model. Defaults to mmdet.

• palette (str) – Color palette used for visualization. The order of priority is palette -> config -> checkpoint. Defaults to ‘none’.

• show_progress (bool) – Control whether to display the progress bar during the inference process. Defaults to True.

postprocess(preds: List[DetDataSample], visualization: Optional[List[ndarray]] = None, return_datasamples: bool = False, print_result: bool = False, no_save_pred: bool = False, pred_out_dir: str = '', **kwargs) → Dict

Process the predictions and visualization results from forward and visualize.

This method should be responsible for the following tasks:

1. Convert datasamples into a json-serializable dict if needed.

2. Pack the predictions and visualization results and return them.

3. Dump or log the predictions.

Parameters

• preds (List[DetDataSample]) – Predictions of the model.

• visualization (Optional[np.ndarray]) – Visualized predictions.

• return_datasamples (bool) – Whether to use Datasample to store inference results. If False, dict will be used.

• print_result (bool) – Whether to print the inference result w/o visualization to the console. Defaults to False.

• no_save_pred (bool) – Whether to force not to save prediction results. Defaults to False.

• pred_out_dir – Dir to save the inference results w/o visualization. If left as empty, no file will be saved. Defaults to ‘’.

Returns

Inference and visualization results with key predictions and visualization.

• visualization (Any): Returned by visualize().

• predictions (dict or DataSample): Returned by

  forward() and processed in postprocess(). If return_datasamples=False, it usually should be a json-serializable dict containing only basic data elements such as strings and numbers.

Return type

dict

pred2dict(data_sample: DetDataSample, pred_out_dir: str = '') → Dict

Extract elements necessary to represent a prediction into a dictionary.

It’s better to contain only basic data elements such as strings and numbers in order to guarantee it’s json-serializable.

Parameters

• data_sample (DetDataSample) – Predictions of the model.

• pred_out_dir – Dir to save the inference results w/o visualization. If left as empty, no file will be saved. Defaults to ‘’.

Returns

Prediction results.

Return type

dict

preprocess(inputs: Union[str, ndarray, Sequence[Union[str, ndarray]]], batch_size: int = 1, **kwargs)

Process the inputs into a model-feedable format.

Customize your preprocess by overriding this method. Preprocess should return an iterable object, of which each item will be used as the input of model.test_step.

BaseInferencer.preprocess will return an iterable chunked data, which will be used in __call__ like this:

def __call__(self, inputs, batch_size=1, **kwargs):
    chunked_data = self.preprocess(inputs, batch_size, **kwargs)
    for batch in chunked_data:
        preds = self.forward(batch, **kwargs)

Parameters

• inputs (InputsType) – Inputs given by user.

• batch_size (int) – batch size. Defaults to 1.

Yields

Any – Data processed by the pipeline and collate_fn.

visualize(inputs: Union[str, ndarray, Sequence[Union[str, ndarray]]], preds: List[DetDataSample], return_vis: bool = False, show: bool = False, wait_time: int = 0, draw_pred: bool = True, pred_score_thr: float = 0.3, no_save_vis: bool = False, img_out_dir: str = '', **kwargs) → Optional[List[ndarray]]

Visualize predictions.

Parameters

• inputs (List[Union[str, np.ndarray]]) – Inputs for the inferencer.

• preds (List[DetDataSample]) – Predictions of the model.

• return_vis (bool) – Whether to return the visualization result. Defaults to False.

• show (bool) – Whether to display the image in a popup window. Defaults to False.

• wait_time (float) – The interval of show (s). Defaults to 0.

• draw_pred (bool) – Whether to draw predicted bounding boxes. Defaults to True.

• pred_score_thr (float) – Minimum score of bboxes to draw. Defaults to 0.3.

• no_save_vis (bool) – Whether to force not to save prediction vis results. Defaults to False.

• img_out_dir (str) – Output directory of visualization results. If left as empty, no file will be saved. Defaults to ‘’.

Returns

Returns visualization results only if applicable.

Return type

List[np.ndarray] or None

async mmdet.apis.async_inference_detector(model, imgs)

Async inference image(s) with the detector.

Parameters

• model (nn.Module) – The loaded detector.

• img (str | ndarray) – Either image files or loaded images.

Returns

Awaitable detection results.

mmdet.apis.inference_detector(model: Module, imgs: Union[str, ndarray, Sequence[str], Sequence[ndarray]], test_pipeline: Optional[Compose] = None, text_prompt: Optional[str] = None, custom_entities: bool = False) → Union[DetDataSample, List[DetDataSample]]

Inference image(s) with the detector.

Parameters

• model (nn.Module) – The loaded detector.

• imgs (str, ndarray, Sequence[str/ndarray]) – Either image files or loaded images.

• test_pipeline (Compose) – Test pipeline.

Returns

If imgs is a list or tuple, the same length list type results will be returned, otherwise return the detection results directly.

Return type

DetDataSample or list[DetDataSample]

mmdet.apis.inference_mot(model: Module, img: ndarray, frame_id: int, video_len: int) → List[DetDataSample]

Inference image(s) with the mot model.

Parameters

• model (nn.Module) – The loaded mot model.

• img (np.ndarray) – Loaded image.

• frame_id (int) – frame id.

• video_len (int) – demo video length

Returns

The tracking data samples.

Return type

SampleList

mmdet.apis.init_detector(config: Union[str, Path, Config], checkpoint: Optional[str] = None, palette: str = 'none', device: str = 'cuda:0', cfg_options: Optional[dict] = None) → Module

Initialize a detector from config file.

Parameters

• config (str, Path, or mmengine.Config) – Config file path, Path, or the config object.

• checkpoint (str, optional) – Checkpoint path. If left as None, the model will not load any weights.

• palette (str) – Color palette used for visualization. If palette is stored in checkpoint, use checkpoint’s palette first, otherwise use externally passed palette. Currently, supports ‘coco’, ‘voc’, ‘citys’ and ‘random’. Defaults to none.

• device (str) – The device where the anchors will be put on. Defaults to cuda:0.

• cfg_options (dict, optional) – Options to override some settings in the used config.

Returns

The constructed detector.

Return type

nn.Module

mmdet.apis.init_track_model(config: Union[str, Config], checkpoint: Optional[str] = None, detector: Optional[str] = None, reid: Optional[str] = None, device: str = 'cuda:0', cfg_options: Optional[dict] = None) → Module

Initialize a model from config file.

Parameters

• config (str or mmengine.Config) – Config file path or the config object.

• checkpoint (Optional[str], optional) – Checkpoint path. Defaults to None.

• detector (Optional[str], optional) – Detector Checkpoint path, use in some tracking algorithms like sort. Defaults to None.

• reid (Optional[str], optional) – Reid checkpoint path. use in some tracking algorithms like sort. Defaults to None.

• device (str, optional) – The device that the model inferences on. Defaults to cuda:0.

• cfg_options (Optional[dict], optional) – Options to override some settings in the used config. Defaults to None.

Returns

The constructed model.

Return type

nn.Module

mmdet.datasets

datasets

class mmdet.datasets.ADE20KInstanceDataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

class mmdet.datasets.ADE20KPanopticDataset(ann_file: str = '', metainfo: Optional[dict] = None, data_root: Optional[str] = None, data_prefix: dict = {'ann': None, 'img': None, 'seg': None}, filter_cfg: Optional[dict] = None, indices: Optional[Union[int, Sequence[int]]] = None, serialize_data: bool = True, pipeline: List[Union[dict, Callable]] = [], test_mode: bool = False, lazy_init: bool = False, max_refetch: int = 1000, backend_args: Optional[dict] = None, **kwargs)

class mmdet.datasets.ADE20KSegDataset(img_suffix='.jpg', seg_map_suffix='.png', return_classes=False, **kwargs)

ADE20K dataset.

In segmentation map annotation for ADE20K, 0 stands for background, which is not included in 150 categories. The img_suffix is fixed to ‘.jpg’, and seg_map_suffix is fixed to ‘.png’.

load_data_list() → List[dict]

Load annotation from directory or annotation file.

Returns

All data info of dataset.

Return type

List[dict]

class mmdet.datasets.AspectRatioBatchSampler(sampler: Sampler, batch_size: int, drop_last: bool = False)

A sampler wrapper for grouping images with similar aspect ratio (< 1 or.

>= 1) into a same batch.

Parameters

• sampler (Sampler) – Base sampler.

• batch_size (int) – Size of mini-batch.

• drop_last (bool) – If True, the sampler will drop the last batch if its size would be less than batch_size.

class mmdet.datasets.BaseDetDataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

Base dataset for detection.

Parameters

• proposal_file (str, optional) – Proposals file path. Defaults to None.

• file_client_args (dict) – Arguments to instantiate the corresponding backend in mmdet <= 3.0.0rc6. Defaults to None.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

• return_classes (bool) – Whether to return class information for open vocabulary-based algorithms. Defaults to False.

• caption_prompt (dict, optional) – Prompt for captioning. Defaults to None.

full_init() → None

Load annotation file and set BaseDataset._fully_initialized to True.

If lazy_init=False, full_init will be called during the instantiation and self._fully_initialized will be set to True. If obj._fully_initialized=False, the class method decorated by force_full_init will call full_init automatically.

Several steps to initialize annotation:

• load_data_list: Load annotations from annotation file.

• load_proposals: Load proposals from proposal file, if self.proposal_file is not None.

• filter data information: Filter annotations according to filter_cfg.

• slice_data: Slice dataset according to self._indices

• serialize_data: Serialize self.data_list if

self.serialize_data is True.

get_cat_ids(idx: int) → List[int]

Get COCO category ids by index.

Parameters

idx (int) – Index of data.

Returns

All categories in the image of specified index.

Return type

List[int]

load_proposals() → None

Load proposals from proposals file.

The proposals_list should be a dict[img_path: proposals] with the same length as data_list. And the proposals should be a dict or InstanceData usually contains following keys.

• bboxes (np.ndarry): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• scores (np.ndarry): Classification scores, has a shape (num_instance, ).

class mmdet.datasets.BaseSegDataset(ann_file: str = '', img_suffix='.jpg', seg_map_suffix='.png', metainfo: Optional[dict] = None, data_root: Optional[str] = None, data_prefix: dict = {'img_path': '', 'seg_map_path': ''}, filter_cfg: Optional[dict] = None, indices: Optional[Union[int, Sequence[int]]] = None, serialize_data: bool = True, pipeline: List[Union[dict, Callable]] = [], test_mode: bool = False, lazy_init: bool = False, use_label_map: bool = False, max_refetch: int = 1000, backend_args: Optional[dict] = None)

Custom dataset for semantic segmentation. An example of file structure is as followed.

├── data
│   ├── my_dataset
│   │   ├── img_dir
│   │   │   ├── train
│   │   │   │   ├── xxx{img_suffix}
│   │   │   │   ├── yyy{img_suffix}
│   │   │   │   ├── zzz{img_suffix}
│   │   │   ├── val
│   │   ├── ann_dir
│   │   │   ├── train
│   │   │   │   ├── xxx{seg_map_suffix}
│   │   │   │   ├── yyy{seg_map_suffix}
│   │   │   │   ├── zzz{seg_map_suffix}
│   │   │   ├── val

The img/gt_semantic_seg pair of BaseSegDataset should be of the same except suffix. A valid img/gt_semantic_seg filename pair should be like xxx{img_suffix} and xxx{seg_map_suffix} (extension is also included in the suffix). If split is given, then xxx is specified in txt file. Otherwise, all files in img_dir/``and ``ann_dir will be loaded. Please refer to docs/en/tutorials/new_dataset.md for more details.

Parameters

• ann_file (str) – Annotation file path. Defaults to ‘’.

• metainfo (dict, optional) – Meta information for dataset, such as specify classes to load. Defaults to None.

• data_root (str, optional) – The root directory for data_prefix and ann_file. Defaults to None.

• data_prefix (dict, optional) – Prefix for training data. Defaults to dict(img_path=None, seg_map_path=None).

• img_suffix (str) – Suffix of images. Default: ‘.jpg’

• seg_map_suffix (str) – Suffix of segmentation maps. Default: ‘.png’

• filter_cfg (dict, optional) – Config for filter data. Defaults to None.

• indices (int or Sequence[int], optional) – Support using first few data in annotation file to facilitate training/testing on a smaller dataset. Defaults to None which means using all data_infos.

• serialize_data (bool, optional) – Whether to hold memory using serialized objects, when enabled, data loader workers can use shared RAM from master process instead of making a copy. Defaults to True.

• pipeline (list, optional) – Processing pipeline. Defaults to [].

• test_mode (bool, optional) – test_mode=True means in test phase. Defaults to False.

• lazy_init (bool, optional) – Whether to load annotation during instantiation. In some cases, such as visualization, only the meta information of the dataset is needed, which is not necessary to load annotation file. Basedataset can skip load annotations to save time by set lazy_init=True. Defaults to False.

• use_label_map (bool, optional) – Whether to use label map. Defaults to False.

• max_refetch (int, optional) – If Basedataset.prepare_data get a None img. The maximum extra number of cycles to get a valid image. Defaults to 1000.

• backend_args (dict, Optional) – Arguments to instantiate a file backend. See https://onedl-mmengine.readthedocs.io/en/latest/api/fileio.htm for details. Defaults to None. Notes: mmcv>=2.0.0rc4 required.

classmethod get_label_map(new_classes: Optional[Sequence] = None) → Optional[Dict]

Require label mapping.

The label_map is a dictionary, its keys are the old label ids and its values are the new label ids, and is used for changing pixel labels in load_annotations. If and only if old classes in cls.METAINFO is not equal to new classes in self._metainfo and nether of them is not None, label_map is not None.

Parameters

new_classes (list, tuple, optional) – The new classes name from metainfo. Default to None.

Returns

The mapping from old classes in cls.METAINFO to

new classes in self._metainfo

Return type

dict, optional

load_data_list() → List[dict]

Load annotation from directory or annotation file.

Returns

All data info of dataset.

Return type

list[dict]

class mmdet.datasets.BaseVideoDataset(*args, backend_args: Optional[dict] = None, **kwargs)

Base video dataset for VID, MOT and VIS tasks.

filter_data() → List[int]

Filter image annotations according to filter_cfg.

Returns

Filtered results.

Return type

list[int]

get_cat_ids(index) → List[int]

Following image detection, we provide this interface function. Get category ids by video index and frame index.

Parameters

index –

The index of the dataset. It support two kinds of inputs: Tuple:

video_idx (int): Index of video. frame_idx (int): Index of frame.

Int: Index of video.

Returns

All categories in the image of specified video index and frame index.

Return type

List[int]

get_len_per_video(idx)

Get length of one video.

Parameters

idx (int) – Index of video.

Returns

The length of the video.

Return type

int (int)

load_data_list() → Tuple[List[dict], List]

Load annotations from an annotation file named as self.ann_file.

Returns

A list of annotation and a list of valid data indices.

Return type

tuple(list[dict], list)

property num_all_imgs

Get the number of all the images in this video dataset.

parse_data_info(raw_data_info: dict) → dict

Parse raw annotation to target format.

Parameters

raw_data_info (dict) – Raw data information loaded from ann_file.

Returns

Parsed annotation.

Return type

dict

prepare_data(idx) → Any

Get date processed by self.pipeline. Note that idx is a video index in default since the base element of video dataset is a video. However, in some cases, we need to specific both the video index and frame index. For example, in traing mode, we may want to sample the specific frames and all the frames must be sampled once in a epoch; in test mode, we may want to output data of a single image rather than the whole video for saving memory.

Parameters

idx (int) – The index of data_info.

Returns

Depends on self.pipeline.

Return type

Any

class mmdet.datasets.CityscapesDataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

Dataset for Cityscapes.

filter_data() → List[dict]

Filter annotations according to filter_cfg.

Returns

Filtered results.

Return type

List[dict]

class mmdet.datasets.ClassAwareSampler(dataset: BaseDataset, seed: Optional[int] = None, num_sample_class: int = 1)

Sampler that restricts data loading to the label of the dataset.

A class-aware sampling strategy to effectively tackle the non-uniform class distribution. The length of the training data is consistent with source data. Simple improvements based on Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks

The implementation logic is referred to https://github.com/Sense-X/TSD/blob/master/mmdet/datasets/samplers/distributed_classaware_sampler.py

Parameters

• dataset – Dataset used for sampling.

• seed (int, optional) – random seed used to shuffle the sampler. This number should be identical across all processes in the distributed group. Defaults to None.

• num_sample_class (int) – The number of samples taken from each per-label list. Defaults to 1.

get_cat2imgs() → Dict[int, list]

Get a dict with class as key and img_ids as values.

Returns

A dict of per-label image list, the item of the dict indicates a label index, corresponds to the image index that contains the label.

Return type

dict[int, list]

set_epoch(epoch: int) → None

Sets the epoch for this sampler.

When shuffle=True, this ensures all replicas use a different random ordering for each epoch. Otherwise, the next iteration of this sampler will yield the same ordering.

Parameters

epoch (int) – Epoch number.

class mmdet.datasets.CocoCaptionDataset(ann_file: Optional[str] = '', metainfo: Optional[Union[Mapping, Config]] = None, data_root: Optional[str] = '', data_prefix: dict = {'img_path': ''}, filter_cfg: Optional[dict] = None, indices: Optional[Union[int, Sequence[int]]] = None, serialize_data: bool = True, pipeline: List[Union[dict, Callable]] = [], test_mode: bool = False, lazy_init: bool = False, max_refetch: int = 1000)

COCO2014 Caption dataset.

load_data_list() → List[dict]

Load data list.

class mmdet.datasets.CocoDataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

Dataset for COCO.

COCOAPI

alias of COCO

filter_data() → List[dict]

Filter annotations according to filter_cfg.

Returns

Filtered results.

Return type

List[dict]

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

Returns

A list of annotation.

Return type

List[dict]

parse_data_info(raw_data_info: dict) → Union[dict, List[dict]]

Parse raw annotation to target format.

Parameters

raw_data_info (dict) – Raw data information load from ann_file

Returns

Parsed annotation.

Return type

Union[dict, List[dict]]

class mmdet.datasets.CocoPanopticDataset(ann_file: str = '', metainfo: Optional[dict] = None, data_root: Optional[str] = None, data_prefix: dict = {'ann': None, 'img': None, 'seg': None}, filter_cfg: Optional[dict] = None, indices: Optional[Union[int, Sequence[int]]] = None, serialize_data: bool = True, pipeline: List[Union[dict, Callable]] = [], test_mode: bool = False, lazy_init: bool = False, max_refetch: int = 1000, backend_args: Optional[dict] = None, **kwargs)

Coco dataset for Panoptic segmentation.

The annotation format is shown as follows. The ann field is optional for testing.

[
    {
        'filename': f'{image_id:012}.png',
        'image_id':9
        'segments_info':
        [
            {
                'id': 8345037, (segment_id in panoptic png,
                                convert from rgb)
                'category_id': 51,
                'iscrowd': 0,
                'bbox': (x1, y1, w, h),
                'area': 24315
            },
            ...
        ]
    },
    ...
]

Parameters

• ann_file (str) – Annotation file path. Defaults to ‘’.

• metainfo (dict, optional) – Meta information for dataset, such as class information. Defaults to None.

• data_root (str, optional) – The root directory for data_prefix and ann_file. Defaults to None.

• data_prefix (dict, optional) – Prefix for training data. Defaults to dict(img=None, ann=None, seg=None). The prefix seg which is for panoptic segmentation map must be not None.

• filter_cfg (dict, optional) – Config for filter data. Defaults to None.

• indices (int or Sequence[int], optional) – Support using first few data in annotation file to facilitate training/testing on a smaller dataset. Defaults to None which means using all data_infos.

• serialize_data (bool, optional) – Whether to hold memory using serialized objects, when enabled, data loader workers can use shared RAM from master process instead of making a copy. Defaults to True.

• pipeline (list, optional) – Processing pipeline. Defaults to [].

• test_mode (bool, optional) – test_mode=True means in test phase. Defaults to False.

• lazy_init (bool, optional) – Whether to load annotation during instantiation. In some cases, such as visualization, only the meta information of the dataset is needed, which is not necessary to load annotation file. Basedataset can skip load annotations to save time by set lazy_init=False. Defaults to False.

• max_refetch (int, optional) – If Basedataset.prepare_data get a None img. The maximum extra number of cycles to get a valid image. Defaults to 1000.

COCOAPI

alias of COCOPanoptic

filter_data() → List[dict]

Filter images too small or without ground truth.

Returns

self.data_list after filtering.

Return type

List[dict]

parse_data_info(raw_data_info: dict) → dict

Parse raw annotation to target format.

Parameters

raw_data_info (dict) – Raw data information load from ann_file.

Returns

Parsed annotation.

Return type

dict

class mmdet.datasets.CocoSegDataset(img_suffix='.jpg', seg_map_suffix='.png', return_classes=False, **kwargs)

COCO dataset.

In segmentation map annotation for COCO. The img_suffix is fixed to ‘.jpg’, and seg_map_suffix is fixed to ‘.png’.

class mmdet.datasets.ConcatDataset(datasets: Sequence[Union[BaseDataset, dict]], lazy_init: bool = False, ignore_keys: Optional[Union[str, List[str]]] = None)

A wrapper of concatenated dataset.

Same as torch.utils.data.dataset.ConcatDataset, support lazy_init and get_dataset_source.

Note

ConcatDataset should not inherit from BaseDataset since get_subset and get_subset_ could produce ambiguous meaning sub-dataset which conflicts with original dataset. If you want to use a sub-dataset of ConcatDataset, you should set indices arguments for wrapped dataset which inherit from BaseDataset.

Parameters

• datasets (Sequence[BaseDataset] or Sequence[dict]) – A list of datasets which will be concatenated.

• lazy_init (bool, optional) – Whether to load annotation during instantiation. Defaults to False.

• ignore_keys (List[str] or str) – Ignore the keys that can be unequal in dataset.metainfo. Defaults to None. New in version 0.3.0.

class mmdet.datasets.CrowdHumanDataset(data_root, ann_file, extra_ann_file=None, **kwargs)

Dataset for CrowdHuman.

Parameters

• data_root (str) – The root directory for data_prefix and ann_file.

• ann_file (str) – Annotation file path.

• extra_ann_file (str | optional) – The path of extra image metas for CrowdHuman. It can be created by CrowdHumanDataset automatically or by tools/misc/get_crowdhuman_id_hw.py manually. Defaults to None.

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

Returns

A list of annotation.

Return type

List[dict]

parse_data_info(raw_data_info: dict) → Union[dict, List[dict]]

Parse raw annotation to target format.

Parameters

raw_data_info (dict) – Raw data information load from ann_file

Returns

Parsed annotation.

Return type

Union[dict, List[dict]]

class mmdet.datasets.CustomSampleSizeSampler(dataset: Sized, dataset_size: Sequence[int], ratio_mode: bool = False, seed: Optional[int] = None, round_up: bool = True)

set_epoch(epoch: int) → None

Sets the epoch for this sampler.

When shuffle=True, this ensures all replicas use a different random ordering for each epoch. Otherwise, the next iteration of this sampler will yield the same ordering.

Parameters

epoch (int) – Epoch number.

class mmdet.datasets.DODDataset(*args, data_root: Optional[str] = '', data_prefix: dict = {'img_path': ''}, **kwargs)

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

If the annotation file does not follow OpenMMLab 2.0 format dataset . The subclass must override this method for load annotations. The meta information of annotation file will be overwritten METAINFO and metainfo argument of constructor.

Returns

A list of annotation.

Return type

list[dict]

class mmdet.datasets.DSDLDetDataset(with_bbox: bool = True, with_polygon: bool = False, with_mask: bool = False, with_imagelevel_label: bool = False, with_hierarchy: bool = False, specific_key_path: dict = {}, pre_transform: dict = {}, **kwargs)

Dataset for dsdl detection.

Parameters

• with_bbox (bool) – Load bbox or not, defaults to be True.

• with_polygon (bool) – Load polygon or not, defaults to be False.

• with_mask (bool) – Load seg map mask or not, defaults to be False.

• with_imagelevel_label (bool) – Load image level label or not, defaults to be False.

• with_hierarchy (bool) – Load hierarchy information or not, defaults to be False.

• specific_key_path (dict) – Path of specific key which can not be loaded by it’s field name.

• pre_transform (dict) – pre-transform functions before loading.

filter_data() → List[dict]

Filter annotations according to filter_cfg.

Returns

Filtered results.

Return type

List[dict]

load_data_list() → List[dict]

Load data info from an dsdl yaml file named as self.ann_file

Returns

A list of data info.

Return type

List[dict]

class mmdet.datasets.DeepFashionDataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

Dataset for DeepFashion.

class mmdet.datasets.Flickr30kDataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

Flickr30K Dataset.

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

If the annotation file does not follow OpenMMLab 2.0 format dataset . The subclass must override this method for load annotations. The meta information of annotation file will be overwritten METAINFO and metainfo argument of constructor.

Returns

A list of annotation.

Return type

list[dict]

class mmdet.datasets.GroupMultiSourceSampler(dataset: BaseDataset, batch_size: int, source_ratio: List[Union[int, float]], shuffle: bool = True, seed: Optional[int] = None)

Group Multi-Source Infinite Sampler.

According to the sampling ratio, sample data from different datasets but the same group to form batches.

Parameters

• dataset (Sized) – The dataset.

• batch_size (int) – Size of mini-batch.

• source_ratio (list[int | float]) – The sampling ratio of different source datasets in a mini-batch.

• shuffle (bool) – Whether shuffle the dataset or not. Defaults to True.

• seed (int, optional) – Random seed. If None, set a random seed. Defaults to None.

mmdet.datasets.LVISDataset

alias of LVISV05Dataset

class mmdet.datasets.LVISV05Dataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

LVIS v0.5 dataset for detection.

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

Returns

A list of annotation.

Return type

List[dict]

class mmdet.datasets.LVISV1Dataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

LVIS v1 dataset for detection.

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

Returns

A list of annotation.

Return type

List[dict]

class mmdet.datasets.MDETRStyleRefCocoDataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

RefCOCO dataset.

Only support evaluation now.

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

If the annotation file does not follow OpenMMLab 2.0 format dataset . The subclass must override this method for load annotations. The meta information of annotation file will be overwritten METAINFO and metainfo argument of constructor.

Returns

A list of annotation.

Return type

list[dict]

class mmdet.datasets.MOTChallengeDataset(visibility_thr: float = -1, *args, **kwargs)

Dataset for MOTChallenge.

Parameters

visibility_thr (float, optional) – The minimum visibility for the objects during training. Default to -1.

parse_data_info(raw_data_info: dict) → Union[dict, List[dict]]

Parse raw annotation to target format. The difference between this function and the one in BaseVideoDataset is that the parsing here adds visibility and mot_conf.

Parameters

raw_data_info (dict) – Raw data information load from ann_file

Returns

Parsed annotation.

Return type

Union[dict, List[dict]]

class mmdet.datasets.MultiImageMixDataset(dataset: Union[BaseDataset, dict], pipeline: Sequence[str], skip_type_keys: Optional[Sequence[str]] = None, max_refetch: int = 15, lazy_init: bool = False)

A wrapper of multiple images mixed dataset.

Suitable for training on multiple images mixed data augmentation like mosaic and mixup. For the augmentation pipeline of mixed image data, the get_indexes method needs to be provided to obtain the image indexes, and you can set skip_flags to change the pipeline running process. At the same time, we provide the dynamic_scale parameter to dynamically change the output image size.

Parameters

• dataset (CustomDataset) – The dataset to be mixed.

• pipeline (Sequence[dict]) – Sequence of transform object or config dict to be composed.

• dynamic_scale (tuple[int], optional) – The image scale can be changed dynamically. Default to None. It is deprecated.

• skip_type_keys (list[str], optional) – Sequence of type string to be skip pipeline. Default to None.

• max_refetch (int) – The maximum number of retry iterations for getting valid results from the pipeline. If the number of iterations is greater than max_refetch, but results is still None, then the iteration is terminated and raise the error. Default: 15.

full_init()

Loop to full_init each dataset.

get_data_info(idx: int) → dict

Get annotation by index.

Parameters

idx (int) – Global index of ConcatDataset.

Returns

The idx-th annotation of the datasets.

Return type

dict

property metainfo: dict

Get the meta information of the multi-image-mixed dataset.

Returns

The meta information of multi-image-mixed dataset.

Return type

dict

update_skip_type_keys(skip_type_keys)

Update skip_type_keys. It is called by an external hook.

Parameters

skip_type_keys (list[str], optional) – Sequence of type string to be skip pipeline.

class mmdet.datasets.MultiSourceSampler(dataset: Sized, batch_size: int, source_ratio: List[Union[int, float]], shuffle: bool = True, seed: Optional[int] = None)

Multi-Source Infinite Sampler.

According to the sampling ratio, sample data from different datasets to form batches.

Parameters

• dataset (Sized) – The dataset.

• batch_size (int) – Size of mini-batch.

• source_ratio (list[int | float]) – The sampling ratio of different source datasets in a mini-batch.

• shuffle (bool) – Whether shuffle the dataset or not. Defaults to True.

• seed (int, optional) – Random seed. If None, set a random seed. Defaults to None.

Examples

>>> dataset_type = 'ConcatDataset'
>>> sub_dataset_type = 'CocoDataset'
>>> data_root = 'data/coco/'
>>> sup_ann = '../coco_semi_annos/instances_train2017.1@10.json'
>>> unsup_ann = '../coco_semi_annos/' \
>>>             'instances_train2017.1@10-unlabeled.json'
>>> dataset = dict(type=dataset_type,
>>>     datasets=[
>>>         dict(
>>>             type=sub_dataset_type,
>>>             data_root=data_root,
>>>             ann_file=sup_ann,
>>>             data_prefix=dict(img='train2017/'),
>>>             filter_cfg=dict(filter_empty_gt=True, min_size=32),
>>>             pipeline=sup_pipeline),
>>>         dict(
>>>             type=sub_dataset_type,
>>>             data_root=data_root,
>>>             ann_file=unsup_ann,
>>>             data_prefix=dict(img='train2017/'),
>>>             filter_cfg=dict(filter_empty_gt=True, min_size=32),
>>>             pipeline=unsup_pipeline),
>>>         ])
>>>     train_dataloader = dict(
>>>         batch_size=5,
>>>         num_workers=5,
>>>         persistent_workers=True,
>>>         sampler=dict(type='MultiSourceSampler',
>>>             batch_size=5, source_ratio=[1, 4]),
>>>         batch_sampler=None,
>>>         dataset=dataset)

set_epoch(epoch: int) → None

Not supported in `epoch-based runner.

class mmdet.datasets.ODVGDataset(*args, data_root: str = '', label_map_file: Optional[str] = None, need_text: bool = True, **kwargs)

Object detection and visual grounding dataset.

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

If the annotation file does not follow OpenMMLab 2.0 format dataset . The subclass must override this method for load annotations. The meta information of annotation file will be overwritten METAINFO and metainfo argument of constructor.

Returns

A list of annotation.

Return type

list[dict]

class mmdet.datasets.Objects365V1Dataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

Objects365 v1 dataset for detection.

COCOAPI

alias of COCO

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

Returns

A list of annotation.

Return type

List[dict]

class mmdet.datasets.Objects365V2Dataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

Objects365 v2 dataset for detection.

COCOAPI

alias of COCO

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

Returns

A list of annotation.

Return type

List[dict]

class mmdet.datasets.OpenImagesChallengeDataset(ann_file: str, **kwargs)

Open Images Challenge dataset for detection.

Parameters

ann_file (str) – Open Images Challenge box annotation in txt format.

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

Returns

A list of annotation.

Return type

List[dict]

class mmdet.datasets.OpenImagesDataset(label_file: str, meta_file: str, hierarchy_file: str, image_level_ann_file: Optional[str] = None, **kwargs)

Open Images dataset for detection.

Parameters

• ann_file (str) – Annotation file path.

• label_file (str) – File path of the label description file that maps the classes names in MID format to their short descriptions.

• meta_file (str) – File path to get image metas.

• hierarchy_file (str) – The file path of the class hierarchy.

• image_level_ann_file (str) – Human-verified image level annotation, which is used in evaluation.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

load_data_list() → List[dict]

Load annotations from an annotation file named as self.ann_file

Returns

A list of annotation.

Return type

List[dict]

class mmdet.datasets.ReIDDataset(triplet_sampler: Optional[dict] = None, *args, **kwargs)

Dataset for ReID.

Parameters

• triplet_sampler (dict, optional) – The sampler for hard mining triplet loss. Defaults to None.

• keys – num_ids (int): The number of person ids. ins_per_id (int): The number of image for each person.

load_data_list() → List[dict]

Load annotations from an annotation file named as ‘’self.ann_file’’.

Returns

A list of annotation.

Return type

list[dict]

prepare_data(idx: int) → Any

Get data processed by ‘’self.pipeline’’.

Parameters

idx (int) – The index of ‘’data_info’’

Returns

Depends on ‘’self.pipeline’’

Return type

Any

triplet_sampling(pos_pid, num_ids: int = 8, ins_per_id: int = 4) → Dict

Triplet sampler for hard mining triplet loss. First, for one pos_pid, random sample ins_per_id images with same person id.

Then, random sample num_ids - 1 images for each negative id. Finally, random sample ins_per_id images for each negative id.

Parameters

• pos_pid (ndarray) – The person id of the anchor.

• num_ids (int) – The number of person ids.

• ins_per_id (int) – The number of images for each person.

Returns

Annotation information of num_ids X ins_per_id images.

Return type

Dict

class mmdet.datasets.RefCocoDataset(data_root: str, ann_file: str, split_file: str, data_prefix: Dict, split: str = 'train', text_mode: str = 'random', **kwargs)

RefCOCO dataset.

The Refcoco and Refcoco+ dataset is based on ReferItGame: Referring to Objects in Photographs of Natural Scenes.

The Refcocog dataset is based on Generation and Comprehension of Unambiguous Object Descriptions.

Parameters

• ann_file (str) – Annotation file path.

• data_root (str) – The root directory for data_prefix and ann_file. Defaults to ‘’.

• data_prefix (str) – Prefix for training data.

• split_file (str) – Split file path.

• split (str) – Split name. Defaults to ‘train’.

• text_mode (str) – Text mode. Defaults to ‘random’.

• **kwargs – Other keyword arguments in BaseDataset.

load_data_list() → List[dict]

Load data list.

class mmdet.datasets.TrackAspectRatioBatchSampler(sampler: Sampler, batch_size: int, drop_last: bool = False)

A sampler wrapper for grouping images with similar aspect ratio (< 1 or.

>= 1) into a same batch.

Parameters

• sampler (Sampler) – Base sampler.

• batch_size (int) – Size of mini-batch.

• drop_last (bool) – If True, the sampler will drop the last batch if its size would be less than batch_size.

class mmdet.datasets.TrackImgSampler(dataset: Sized, seed: Optional[int] = None)

Sampler that providing image-level sampling outputs for video datasets in tracking tasks. It could be both used in both distributed and non-distributed environment. If using the default sampler in pytorch, the subsequent data receiver will get one video, which is not desired in some cases: (Take a non-distributed environment as an example) 1. In test mode, we want only one image is fed into the data pipeline. This is in consideration of memory usage since feeding the whole video commonly requires a large amount of memory (>=20G on MOTChallenge17 dataset), which is not available in some machines. 2. In training mode, we may want to make sure all the images in one video are randomly sampled once in one epoch and this can not be guaranteed in the default sampler in pytorch.

Parameters

• dataset (Sized) – Dataset used for sampling.

• seed (int, optional) – random seed used to shuffle the sampler. This number should be identical across all processes in the distributed group. Defaults to None.

class mmdet.datasets.V3DetDataset(*args, metainfo: Optional[dict] = None, data_root: str = '', label_file='annotations/category_name_13204_v3det_2023_v1.txt', **kwargs)

Dataset for V3Det.

class mmdet.datasets.VOCDataset(**kwargs)

Dataset for PASCAL VOC.

class mmdet.datasets.WIDERFaceDataset(img_subdir: str = 'JPEGImages', ann_subdir: str = 'Annotations', **kwargs)

Reader for the WIDER Face dataset in PASCAL VOC format.

Conversion scripts can be found in https://github.com/sovrasov/wider-face-pascal-voc-annotations

load_data_list() → List[dict]

Load annotation from XML style ann_file.

Returns

Annotation info from XML file.

Return type

list[dict]

parse_data_info(img_info: dict) → Union[dict, List[dict]]

Parse raw annotation to target format.

Parameters

img_info (dict) – Raw image information, usually it includes img_id, file_name, and xml_path.

Returns

Parsed annotation.

Return type

Union[dict, List[dict]]

class mmdet.datasets.XMLDataset(img_subdir: str = 'JPEGImages', ann_subdir: str = 'Annotations', **kwargs)

XML dataset for detection.

Parameters

• img_subdir (str) – Subdir where images are stored. Default: JPEGImages.

• ann_subdir (str) – Subdir where annotations are. Default: Annotations.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

property bbox_min_size: Optional[int]

Return the minimum size of bounding boxes in the images.

filter_data() → List[dict]

Filter annotations according to filter_cfg.

Returns

Filtered results.

Return type

List[dict]

load_data_list() → List[dict]

Load annotation from XML style ann_file.

Returns

Annotation info from XML file.

Return type

list[dict]

parse_data_info(img_info: dict) → Union[dict, List[dict]]

Parse raw annotation to target format.

Parameters

img_info (dict) – Raw image information, usually it includes img_id, file_name, and xml_path.

Returns

Parsed annotation.

Return type

Union[dict, List[dict]]

property sub_data_root: str

Return the sub data root.

class mmdet.datasets.YouTubeVISDataset(dataset_version: str, *args, **kwargs)

YouTube VIS dataset for video instance segmentation.

Parameters

dataset_version (str) – Select dataset year version.

classmethod set_dataset_classes(dataset_version: str) → None

Pass the category of the corresponding year to metainfo.

Parameters

dataset_version (str) – Select dataset year version.

mmdet.datasets.get_loading_pipeline(pipeline)

Only keep loading image and annotations related configuration.

Parameters

pipeline (list[dict]) – Data pipeline configs.

Returns

The new pipeline list with only keep

loading image and annotations related configuration.

Return type

list[dict]

Examples

>>> pipelines = [
...    dict(type='LoadImageFromFile'),
...    dict(type='LoadAnnotations', with_bbox=True),
...    dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
...    dict(type='RandomFlip', flip_ratio=0.5),
...    dict(type='Normalize', **img_norm_cfg),
...    dict(type='Pad', size_divisor=32),
...    dict(type='DefaultFormatBundle'),
...    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
...    ]
>>> expected_pipelines = [
...    dict(type='LoadImageFromFile'),
...    dict(type='LoadAnnotations', with_bbox=True)
...    ]
>>> assert expected_pipelines == get_loading_pipeline(pipelines)

class mmdet.datasets.iSAIDDataset(*args, seg_map_suffix: str = '.png', proposal_file: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, return_classes: bool = False, caption_prompt: Optional[dict] = None, **kwargs)

Dataset for iSAID instance segmentation.

iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images.

For more detail, please refer to “projects/iSAID/README.md”

api_wrappers

class mmdet.datasets.api_wrappers.COCO(*args: Any, **kwargs: Any)

This class is almost the same as official pycocotools package.

It implements some snake case function aliases. So that the COCO class has the same interface as LVIS class.

class mmdet.datasets.api_wrappers.COCOPanoptic(*args: Any, **kwargs: Any)

This wrapper is for loading the panoptic style annotation file.

The format is shown in the CocoPanopticDataset class.

Parameters

annotation_file (str, optional) – Path of annotation file. Defaults to None.

createIndex() → None

Create index.

load_anns(ids: Union[List[int], int] = []) → Optional[List[dict]]

Load anns with the specified ids.

self.anns is a list of annotation lists instead of a list of annotations.

Parameters

ids (Union[List[int], int]) – Integer ids specifying anns.

Returns

Loaded ann objects.

Return type

anns (List[dict], optional)

class mmdet.datasets.api_wrappers.COCOevalMP(*args: Any, **kwargs: Any)

evaluate()

Run per image evaluation on given images and store results (a list of dict) in self.evalImgs.

Returns

None

summarize()

Compute and display summary metrics for evaluation results.

Note this function can only be applied on the default parameter setting

samplers

class mmdet.datasets.samplers.AspectRatioBatchSampler(sampler: Sampler, batch_size: int, drop_last: bool = False)

A sampler wrapper for grouping images with similar aspect ratio (< 1 or.

>= 1) into a same batch.

Parameters

• sampler (Sampler) – Base sampler.

• batch_size (int) – Size of mini-batch.

• drop_last (bool) – If True, the sampler will drop the last batch if its size would be less than batch_size.

class mmdet.datasets.samplers.ClassAwareSampler(dataset: BaseDataset, seed: Optional[int] = None, num_sample_class: int = 1)

Sampler that restricts data loading to the label of the dataset.

A class-aware sampling strategy to effectively tackle the non-uniform class distribution. The length of the training data is consistent with source data. Simple improvements based on Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks

The implementation logic is referred to https://github.com/Sense-X/TSD/blob/master/mmdet/datasets/samplers/distributed_classaware_sampler.py

Parameters

• dataset – Dataset used for sampling.

• seed (int, optional) – random seed used to shuffle the sampler. This number should be identical across all processes in the distributed group. Defaults to None.

• num_sample_class (int) – The number of samples taken from each per-label list. Defaults to 1.

get_cat2imgs() → Dict[int, list]

Get a dict with class as key and img_ids as values.

Returns

A dict of per-label image list, the item of the dict indicates a label index, corresponds to the image index that contains the label.

Return type

dict[int, list]

set_epoch(epoch: int) → None

Sets the epoch for this sampler.

When shuffle=True, this ensures all replicas use a different random ordering for each epoch. Otherwise, the next iteration of this sampler will yield the same ordering.

Parameters

epoch (int) – Epoch number.

class mmdet.datasets.samplers.CustomSampleSizeSampler(dataset: Sized, dataset_size: Sequence[int], ratio_mode: bool = False, seed: Optional[int] = None, round_up: bool = True)

set_epoch(epoch: int) → None

Sets the epoch for this sampler.

When shuffle=True, this ensures all replicas use a different random ordering for each epoch. Otherwise, the next iteration of this sampler will yield the same ordering.

Parameters

epoch (int) – Epoch number.

class mmdet.datasets.samplers.GroupMultiSourceSampler(dataset: BaseDataset, batch_size: int, source_ratio: List[Union[int, float]], shuffle: bool = True, seed: Optional[int] = None)

Group Multi-Source Infinite Sampler.

According to the sampling ratio, sample data from different datasets but the same group to form batches.

Parameters

• dataset (Sized) – The dataset.

• batch_size (int) – Size of mini-batch.

• source_ratio (list[int | float]) – The sampling ratio of different source datasets in a mini-batch.

• shuffle (bool) – Whether shuffle the dataset or not. Defaults to True.

• seed (int, optional) – Random seed. If None, set a random seed. Defaults to None.

class mmdet.datasets.samplers.MultiDataAspectRatioBatchSampler(sampler: Sampler, batch_size: Sequence[int], num_datasets: int, drop_last: bool = True)

A sampler wrapper for grouping images with similar aspect ratio (< 1 or.

>= 1) into a same batch for multi-source datasets.

Parameters

• sampler (Sampler) – Base sampler.

• batch_size (Sequence(int)) – Size of mini-batch for multi-source

• datasets. –

• num_datasets (int) – Number of multi-source datasets.

• drop_last (bool) – If True, the sampler will drop the last batch if

• batch_size. (its size would be less than) –

class mmdet.datasets.samplers.MultiDataSampler(dataset: Sized, dataset_ratio: Sequence[int], seed: Optional[int] = None, round_up: bool = True)

The default data sampler for both distributed and non-distributed environment.

It has several differences from the PyTorch DistributedSampler as below:

1. This sampler supports non-distributed environment.

2. The round up behaviors are a little different.

   • If round_up=True, this sampler will add extra samples to make the number of samples is evenly divisible by the world size. And this behavior is the same as the DistributedSampler with drop_last=False.

   • If round_up=False, this sampler won’t remove or add any samples while the DistributedSampler with drop_last=True will remove tail samples.

Parameters

• dataset (Sized) – The dataset.

• dataset_ratio (Sequence(int)) –

• seed (int, optional) – Random seed used to shuffle the sampler if shuffle=True. This number should be identical across all processes in the distributed group. Defaults to None.

• round_up (bool) – Whether to add extra samples to make the number of samples evenly divisible by the world size. Defaults to True.

set_epoch(epoch: int) → None

Sets the epoch for this sampler.

When shuffle=True, this ensures all replicas use a different random ordering for each epoch. Otherwise, the next iteration of this sampler will yield the same ordering.

Parameters

epoch (int) – Epoch number.

class mmdet.datasets.samplers.MultiSourceSampler(dataset: Sized, batch_size: int, source_ratio: List[Union[int, float]], shuffle: bool = True, seed: Optional[int] = None)

Multi-Source Infinite Sampler.

According to the sampling ratio, sample data from different datasets to form batches.

Parameters

• dataset (Sized) – The dataset.

• batch_size (int) – Size of mini-batch.

• source_ratio (list[int | float]) – The sampling ratio of different source datasets in a mini-batch.

• shuffle (bool) – Whether shuffle the dataset or not. Defaults to True.

• seed (int, optional) – Random seed. If None, set a random seed. Defaults to None.

Examples

>>> dataset_type = 'ConcatDataset'
>>> sub_dataset_type = 'CocoDataset'
>>> data_root = 'data/coco/'
>>> sup_ann = '../coco_semi_annos/instances_train2017.1@10.json'
>>> unsup_ann = '../coco_semi_annos/' \
>>>             'instances_train2017.1@10-unlabeled.json'
>>> dataset = dict(type=dataset_type,
>>>     datasets=[
>>>         dict(
>>>             type=sub_dataset_type,
>>>             data_root=data_root,
>>>             ann_file=sup_ann,
>>>             data_prefix=dict(img='train2017/'),
>>>             filter_cfg=dict(filter_empty_gt=True, min_size=32),
>>>             pipeline=sup_pipeline),
>>>         dict(
>>>             type=sub_dataset_type,
>>>             data_root=data_root,
>>>             ann_file=unsup_ann,
>>>             data_prefix=dict(img='train2017/'),
>>>             filter_cfg=dict(filter_empty_gt=True, min_size=32),
>>>             pipeline=unsup_pipeline),
>>>         ])
>>>     train_dataloader = dict(
>>>         batch_size=5,
>>>         num_workers=5,
>>>         persistent_workers=True,
>>>         sampler=dict(type='MultiSourceSampler',
>>>             batch_size=5, source_ratio=[1, 4]),
>>>         batch_sampler=None,
>>>         dataset=dataset)

set_epoch(epoch: int) → None

Not supported in `epoch-based runner.

class mmdet.datasets.samplers.TrackAspectRatioBatchSampler(sampler: Sampler, batch_size: int, drop_last: bool = False)

A sampler wrapper for grouping images with similar aspect ratio (< 1 or.

>= 1) into a same batch.

Parameters

• sampler (Sampler) – Base sampler.

• batch_size (int) – Size of mini-batch.

• drop_last (bool) – If True, the sampler will drop the last batch if its size would be less than batch_size.

class mmdet.datasets.samplers.TrackImgSampler(dataset: Sized, seed: Optional[int] = None)

Sampler that providing image-level sampling outputs for video datasets in tracking tasks. It could be both used in both distributed and non-distributed environment. If using the default sampler in pytorch, the subsequent data receiver will get one video, which is not desired in some cases: (Take a non-distributed environment as an example) 1. In test mode, we want only one image is fed into the data pipeline. This is in consideration of memory usage since feeding the whole video commonly requires a large amount of memory (>=20G on MOTChallenge17 dataset), which is not available in some machines. 2. In training mode, we may want to make sure all the images in one video are randomly sampled once in one epoch and this can not be guaranteed in the default sampler in pytorch.

Parameters

• dataset (Sized) – Dataset used for sampling.

• seed (int, optional) – random seed used to shuffle the sampler. This number should be identical across all processes in the distributed group. Defaults to None.

transforms

class mmdet.datasets.transforms.Albu(transforms: List[dict], bbox_params: Optional[dict] = None, keymap: Optional[dict] = None, skip_img_without_anno: bool = False)

Albumentation augmentation.

Adds custom transformations from Albumentations library. Please, visit https://albumentations.readthedocs.io to get more information.

Required Keys:

• img (np.uint8)

• gt_bboxes (HorizontalBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

Modified Keys:

• img (np.uint8)

• gt_bboxes (HorizontalBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• img_shape (tuple)

An example of transforms is as followed:

[
    dict(
        type='ShiftScaleRotate',
        shift_limit=0.0625,
        scale_limit=0.0,
        rotate_limit=0,
        interpolation=1,
        p=0.5),
    dict(
        type='RandomBrightnessContrast',
        brightness_limit=[0.1, 0.3],
        contrast_limit=[0.1, 0.3],
        p=0.2),
    dict(type='ChannelShuffle', p=0.1),
    dict(
        type='OneOf',
        transforms=[
            dict(type='Blur', blur_limit=3, p=1.0),
            dict(type='MedianBlur', blur_limit=3, p=1.0)
        ],
        p=0.1),
]

Parameters

• transforms (list[dict]) – A list of albu transformations

• bbox_params (dict, optional) – Bbox_params for albumentation Compose

• keymap (dict, optional) – Contains {‘input key’:’albumentation-style key’}

• skip_img_without_anno (bool) – Whether to skip the image if no ann left after aug. Defaults to False.

albu_builder(cfg: dict) → <module 'albumentations' from '/home/docs/checkouts/readthedocs.org/user_builds/onedl-mmdetection/envs/5/lib/python3.10/site-packages/albumentations/__init__.py'>

Import a module from albumentations.

It inherits some of build_from_cfg() logic.

Parameters

cfg (dict) – Config dict. It should at least contain the key “type”.

Returns

The constructed object.

Return type

obj

static mapper(d: dict, keymap: dict) → dict

Dictionary mapper. Renames keys according to keymap provided.

Parameters

• d (dict) – old dict

• keymap (dict) – {‘old_key’:’new_key’}

Returns

new dict.

Return type

dict

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.AutoAugment(policies: List[List[Union[ConfigDict, dict]]] = [[{'type': 'Equalize', 'prob': 0.8, 'level': 1}, {'type': 'ShearY', 'prob': 0.8, 'level': 4}], [{'type': 'Color', 'prob': 0.4, 'level': 9}, {'type': 'Equalize', 'prob': 0.6, 'level': 3}], [{'type': 'Color', 'prob': 0.4, 'level': 1}, {'type': 'Rotate', 'prob': 0.6, 'level': 8}], [{'type': 'Solarize', 'prob': 0.8, 'level': 3}, {'type': 'Equalize', 'prob': 0.4, 'level': 7}], [{'type': 'Solarize', 'prob': 0.4, 'level': 2}, {'type': 'Solarize', 'prob': 0.6, 'level': 2}], [{'type': 'Color', 'prob': 0.2, 'level': 0}, {'type': 'Equalize', 'prob': 0.8, 'level': 8}], [{'type': 'Equalize', 'prob': 0.4, 'level': 8}, {'type': 'SolarizeAdd', 'prob': 0.8, 'level': 3}], [{'type': 'ShearX', 'prob': 0.2, 'level': 9}, {'type': 'Rotate', 'prob': 0.6, 'level': 8}], [{'type': 'Color', 'prob': 0.6, 'level': 1}, {'type': 'Equalize', 'prob': 1.0, 'level': 2}], [{'type': 'Invert', 'prob': 0.4, 'level': 9}, {'type': 'Rotate', 'prob': 0.6, 'level': 0}], [{'type': 'Equalize', 'prob': 1.0, 'level': 9}, {'type': 'ShearY', 'prob': 0.6, 'level': 3}], [{'type': 'Color', 'prob': 0.4, 'level': 7}, {'type': 'Equalize', 'prob': 0.6, 'level': 0}], [{'type': 'Posterize', 'prob': 0.4, 'level': 6}, {'type': 'AutoContrast', 'prob': 0.4, 'level': 7}], [{'type': 'Solarize', 'prob': 0.6, 'level': 8}, {'type': 'Color', 'prob': 0.6, 'level': 9}], [{'type': 'Solarize', 'prob': 0.2, 'level': 4}, {'type': 'Rotate', 'prob': 0.8, 'level': 9}], [{'type': 'Rotate', 'prob': 1.0, 'level': 7}, {'type': 'TranslateY', 'prob': 0.8, 'level': 9}], [{'type': 'ShearX', 'prob': 0.0, 'level': 0}, {'type': 'Solarize', 'prob': 0.8, 'level': 4}], [{'type': 'ShearY', 'prob': 0.8, 'level': 0}, {'type': 'Color', 'prob': 0.6, 'level': 4}], [{'type': 'Color', 'prob': 1.0, 'level': 0}, {'type': 'Rotate', 'prob': 0.6, 'level': 2}], [{'type': 'Equalize', 'prob': 0.8, 'level': 4}, {'type': 'Equalize', 'prob': 0.0, 'level': 8}], [{'type': 'Equalize', 'prob': 1.0, 'level': 4}, {'type': 'AutoContrast', 'prob': 0.6, 'level': 2}], [{'type': 'ShearY', 'prob': 0.4, 'level': 7}, {'type': 'SolarizeAdd', 'prob': 0.6, 'level': 7}], [{'type': 'Posterize', 'prob': 0.8, 'level': 2}, {'type': 'Solarize', 'prob': 0.6, 'level': 10}], [{'type': 'Solarize', 'prob': 0.6, 'level': 8}, {'type': 'Equalize', 'prob': 0.6, 'level': 1}], [{'type': 'Color', 'prob': 0.8, 'level': 6}, {'type': 'Rotate', 'prob': 0.4, 'level': 5}]], prob: Optional[List[float]] = None)

Auto augmentation.

This data augmentation is proposed in AutoAugment: Learning Augmentation Policies from Data and in Learning Data Augmentation Strategies for Object Detection.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_ignore_flags (bool) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• img_shape

• gt_bboxes

• gt_bboxes_labels

• gt_masks

• gt_ignore_flags

• gt_seg_map

Added Keys:

• homography_matrix

Parameters

• policies (List[List[Union[dict, ConfigDict]]]) – The policies of auto augmentation.Each policy in policies is a specific augmentation policy, and is composed by several augmentations. When AutoAugment is called, a random policy in policies will be selected to augment images. Defaults to policy_v0().

• prob (list[float], optional) – The probabilities associated with each policy. The length should be equal to the policy number and the sum should be 1. If not given, a uniform distribution will be assumed. Defaults to None.

Examples

>>> policies = [
>>>     [
>>>         dict(type='Sharpness', prob=0.0, level=8),
>>>         dict(type='ShearX', prob=0.4, level=0,)
>>>     ],
>>>     [
>>>         dict(type='Rotate', prob=0.6, level=10),
>>>         dict(type='Color', prob=1.0, level=6)
>>>     ]
>>> ]
>>> augmentation = AutoAugment(policies)
>>> img = np.ones(100, 100, 3)
>>> gt_bboxes = np.ones(10, 4)
>>> results = dict(img=img, gt_bboxes=gt_bboxes)
>>> results = augmentation(results)

class mmdet.datasets.transforms.AutoContrast(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.1, max_mag: float = 1.9)

Auto adjust image contrast.

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing AutoContrast should be in range [0, 1]. Defaults to 1.0.

• level (int, optional) – No use for AutoContrast transformation. Defaults to None.

• min_mag (float) – No use for AutoContrast transformation. Defaults to 0.1.

• max_mag (float) – No use for AutoContrast transformation. Defaults to 1.9.

class mmdet.datasets.transforms.BaseFrameSample(collect_video_keys: List[str] = ['video_id', 'video_length'])

Directly get the key frame, no reference frames.

Parameters

collect_video_keys (list[str]) – The keys of video info to be collected.

prepare_data(video_infos: dict, sampled_inds: List[int]) → Dict[str, List]

Prepare data for the subsequent pipeline.

Parameters

• video_infos (dict) – The whole video information.

• sampled_inds (list[int]) – The sampled frame indices.

Returns

The processed data information.

Return type

dict

transform(video_infos: dict) → Optional[Dict[str, List]]

Transform the video information.

Parameters

video_infos (dict) – The whole video information.

Returns

The data information of the key frames.

Return type

dict

class mmdet.datasets.transforms.Brightness(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.1, max_mag: float = 1.9)

Adjust the brightness of the image. A magnitude=0 gives a black image, whereas magnitude=1 gives the original image. The bboxes, masks and segmentations are not modified.

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing Brightness transformation. Defaults to 1.0.

• level (int, optional) – Should be in range [0,_MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum magnitude for Brightness transformation. Defaults to 0.1.

• max_mag (float) – The maximum magnitude for Brightness transformation. Defaults to 1.9.

class mmdet.datasets.transforms.CachedMixUp(img_scale: Tuple[int, int] = (640, 640), ratio_range: Tuple[float, float] = (0.5, 1.5), flip_ratio: float = 0.5, pad_val: float = 114.0, max_iters: int = 15, bbox_clip_border: bool = True, max_cached_images: int = 20, random_pop: bool = True, prob: float = 1.0)

Cached mixup data augmentation.

                    mixup transform
           +------------------------------+
           | mixup image   |              |
           |      +--------|--------+     |
           |      |        |        |     |
           |---------------+        |     |
           |      |                 |     |
           |      |      image      |     |
           |      |                 |     |
           |      |                 |     |
           |      |-----------------+     |
           |             pad              |
           +------------------------------+

The cached mixup transform steps are as follows:

   1. Append the results from the last transform into the cache.
   2. Another random image is picked from the cache and embedded in
      the top left patch(after padding and resizing)
   3. The target of mixup transform is the weighted average of mixup
      image and origin image.

Required Keys:

• img

• gt_bboxes (np.float32) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_ignore_flags (bool) (optional)

• mix_results (List[dict])

Modified Keys:

• img

• img_shape

• gt_bboxes (optional)

• gt_bboxes_labels (optional)

• gt_ignore_flags (optional)

Parameters

• img_scale (Sequence[int]) – Image output size after mixup pipeline. The shape order should be (width, height). Defaults to (640, 640).

• ratio_range (Sequence[float]) – Scale ratio of mixup image. Defaults to (0.5, 1.5).

• flip_ratio (float) – Horizontal flip ratio of mixup image. Defaults to 0.5.

• pad_val (int) – Pad value. Defaults to 114.

• max_iters (int) – The maximum number of iterations. If the number of iterations is greater than max_iters, but gt_bbox is still empty, then the iteration is terminated. Defaults to 15.

• bbox_clip_border (bool, optional) – Whether to clip the objects outside the border of the image. In some dataset like MOT17, the gt bboxes are allowed to cross the border of images. Therefore, we don’t need to clip the gt bboxes in these cases. Defaults to True.

• max_cached_images (int) – The maximum length of the cache. The larger the cache, the stronger the randomness of this transform. As a rule of thumb, providing 10 caches for each image suffices for randomness. Defaults to 20.

• random_pop (bool) – Whether to randomly pop a result from the cache when the cache is full. If set to False, use FIFO popping method. Defaults to True.

• prob (float) – Probability of applying this transformation. Defaults to 1.0.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.CachedMosaic(*args, max_cached_images: int = 40, random_pop: bool = True, **kwargs)

Cached mosaic augmentation.

Cached mosaic transform will random select images from the cache and combine them into one output image.

                   mosaic transform
                      center_x
           +------------------------------+
           |       pad        |  pad      |
           |      +-----------+           |
           |      |           |           |
           |      |  image1   |--------+  |
           |      |           |        |  |
           |      |           | image2 |  |
center_y   |----+-------------+-----------|
           |    |   cropped   |           |
           |pad |   image3    |  image4   |
           |    |             |           |
           +----|-------------+-----------+
                |             |
                +-------------+

The cached mosaic transform steps are as follows:

    1. Append the results from the last transform into the cache.
    2. Choose the mosaic center as the intersections of 4 images
    3. Get the left top image according to the index, and randomly
       sample another 3 images from the result cache.
    4. Sub image will be cropped if image is larger than mosaic patch

Required Keys:

• img

• gt_bboxes (np.float32) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_ignore_flags (bool) (optional)

Modified Keys:

• img

• img_shape

• gt_bboxes (optional)

• gt_bboxes_labels (optional)

• gt_ignore_flags (optional)

Parameters

• img_scale (Sequence[int]) – Image size before mosaic pipeline of single image. The shape order should be (width, height). Defaults to (640, 640).

• center_ratio_range (Sequence[float]) – Center ratio range of mosaic output. Defaults to (0.5, 1.5).

• bbox_clip_border (bool, optional) – Whether to clip the objects outside the border of the image. In some dataset like MOT17, the gt bboxes are allowed to cross the border of images. Therefore, we don’t need to clip the gt bboxes in these cases. Defaults to True.

• pad_val (int) – Pad value. Defaults to 114.

• prob (float) – Probability of applying this transformation. Defaults to 1.0.

• max_cached_images (int) – The maximum length of the cache. The larger the cache, the stronger the randomness of this transform. As a rule of thumb, providing 10 caches for each image suffices for randomness. Defaults to 40.

• random_pop (bool) – Whether to randomly pop a result from the cache when the cache is full. If set to False, use FIFO popping method. Defaults to True.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.Color(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.1, max_mag: float = 1.9)

Adjust the color balance of the image, in a manner similar to the controls on a colour TV set. A magnitude=0 gives a black & white image, whereas magnitude=1 gives the original image. The bboxes, masks and segmentations are not modified.

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing Color transformation. Defaults to 1.0.

• level (int, optional) – Should be in range [0,_MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum magnitude for Color transformation. Defaults to 0.1.

• max_mag (float) – The maximum magnitude for Color transformation. Defaults to 1.9.

class mmdet.datasets.transforms.ColorTransform(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.1, max_mag: float = 1.9)

Base class for color transformations. All color transformations need to inherit from this base class. ColorTransform unifies the class attributes and class functions of color transformations (Color, Brightness, Contrast, Sharpness, Solarize, SolarizeAdd, Equalize, AutoContrast, Invert, and Posterize), and only distort color channels, without impacting the locations of the instances.

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing the geometric transformation and should be in range [0, 1]. Defaults to 1.0.

• level (int, optional) – The level should be in range [0, _MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum magnitude for color transformation. Defaults to 0.1.

• max_mag (float) – The maximum magnitude for color transformation. Defaults to 1.9.

transform(results: dict) → dict

Transform function for images.

Parameters

results (dict) – Result dict from loading pipeline.

Returns

Transformed results.

Return type

dict

class mmdet.datasets.transforms.Contrast(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.1, max_mag: float = 1.9)

Control the contrast of the image. A magnitude=0 gives a gray image, whereas magnitude=1 gives the original imageThe bboxes, masks and segmentations are not modified.

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing Contrast transformation. Defaults to 1.0.

• level (int, optional) – Should be in range [0,_MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum magnitude for Contrast transformation. Defaults to 0.1.

• max_mag (float) – The maximum magnitude for Contrast transformation. Defaults to 1.9.

class mmdet.datasets.transforms.CopyPaste(max_num_pasted: int = 100, bbox_occluded_thr: int = 10, mask_occluded_thr: int = 300, selected: bool = True, paste_by_box: bool = False)

Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation The simple copy-paste transform steps are as follows:

1. The destination image is already resized with aspect ratio kept, cropped and padded.

2. Randomly select a source image, which is also already resized with aspect ratio kept, cropped and padded in a similar way as the destination image.

3. Randomly select some objects from the source image.

4. Paste these source objects to the destination image directly, due to the source and destination image have the same size.

5. Update object masks of the destination image, for some origin objects may be occluded.

6. Generate bboxes from the updated destination masks and filter some objects which are totally occluded, and adjust bboxes which are partly occluded.

7. Append selected source bboxes, masks, and labels.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_ignore_flags (bool) (optional)

• gt_masks (BitmapMasks) (optional)

Modified Keys:

• img

• gt_bboxes (optional)

• gt_bboxes_labels (optional)

• gt_ignore_flags (optional)

• gt_masks (optional)

Parameters

• max_num_pasted (int) – The maximum number of pasted objects. Defaults to 100.

• bbox_occluded_thr (int) – The threshold of occluded bbox. Defaults to 10.

• mask_occluded_thr (int) – The threshold of occluded mask. Defaults to 300.

• selected (bool) – Whether select objects or not. If select is False, all objects of the source image will be pasted to the destination image. Defaults to True.

• paste_by_box (bool) – Whether use boxes as masks when masks are not available. Defaults to False.

get_gt_masks(results: dict) → BitmapMasks

Get gt_masks originally or generated based on bboxes.

If gt_masks is not contained in results, it will be generated based on gt_bboxes. :param results: Result dict. :type results: dict

Returns

gt_masks, originally or generated based on bboxes.

Return type

BitmapMasks

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.CutOut(n_holes: Union[int, Tuple[int, int]], cutout_shape: Optional[Union[Tuple[int, int], List[Tuple[int, int]]]] = None, cutout_ratio: Optional[Union[Tuple[float, float], List[Tuple[float, float]]]] = None, fill_in: Union[Tuple[float, float, float], Tuple[int, int, int]] = (0, 0, 0))

CutOut operation.

Randomly drop some regions of image used in Cutout.

Required Keys:

• img

Modified Keys:

• img

Parameters

• n_holes (int or tuple[int, int]) – Number of regions to be dropped. If it is given as a list, number of holes will be randomly selected from the closed interval [n_holes[0], n_holes[1]].

• cutout_shape (tuple[int, int] or list[tuple[int, int]], optional) – The candidate shape of dropped regions. It can be tuple[int, int] to use a fixed cutout shape, or list[tuple[int, int]] to randomly choose shape from the list. Defaults to None.

• (tuple[float (cutout_ratio) – optional): The candidate ratio of dropped regions. It can be tuple[float, float] to use a fixed ratio or list[tuple[float, float]] to randomly choose ratio from the list. Please note that cutout_shape and cutout_ratio cannot be both given at the same time. Defaults to None.

• list[tuple[float (float] or) – optional): The candidate ratio of dropped regions. It can be tuple[float, float] to use a fixed ratio or list[tuple[float, float]] to randomly choose ratio from the list. Please note that cutout_shape and cutout_ratio cannot be both given at the same time. Defaults to None.

• float]] – optional): The candidate ratio of dropped regions. It can be tuple[float, float] to use a fixed ratio or list[tuple[float, float]] to randomly choose ratio from the list. Please note that cutout_shape and cutout_ratio cannot be both given at the same time. Defaults to None.

:paramoptional): The candidate ratio of dropped regions. It can be

tuple[float, float] to use a fixed ratio or list[tuple[float, float]] to randomly choose ratio from the list. Please note that cutout_shape and cutout_ratio cannot be both given at the same time. Defaults to None.

Parameters

fill_in (tuple[float, float, float] or tuple[int, int, int]) – The value of pixel to fill in the dropped regions. Defaults to (0, 0, 0).

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.Equalize(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.1, max_mag: float = 1.9)

Equalize the image histogram. The bboxes, masks and segmentations are not modified.

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing Equalize transformation. Defaults to 1.0.

• level (int, optional) – No use for Equalize transformation. Defaults to None.

• min_mag (float) – No use for Equalize transformation. Defaults to 0.1.

• max_mag (float) – No use for Equalize transformation. Defaults to 1.9.

class mmdet.datasets.transforms.Expand(mean: Sequence[Union[float, int]] = (0, 0, 0), to_rgb: bool = True, ratio_range: Sequence[Union[float, int]] = (1, 4), seg_ignore_label: Optional[int] = None, prob: float = 0.5)

Random expand the image & bboxes & masks & segmentation map.

Randomly place the original image on a canvas of ratio x original image size filled with mean values. The ratio is in the range of ratio_range.

Required Keys:

• img

• img_shape

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• img_shape

• gt_bboxes

• gt_masks

• gt_seg_map

Parameters

• mean (sequence) – mean value of dataset.

• to_rgb (bool) – if need to convert the order of mean to align with RGB.

• ratio_range (sequence)) – range of expand ratio.

• seg_ignore_label (int) – label of ignore segmentation map.

• prob (float) – probability of applying this transformation

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.FilterAnnotations(min_gt_bbox_wh: Tuple[int, int] = (1, 1), min_gt_mask_area: int = 1, by_box: bool = True, by_mask: bool = False, keep_empty: bool = True)

Filter invalid annotations.

Required Keys:

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_ignore_flags (bool) (optional)

Modified Keys:

• gt_bboxes (optional)

• gt_bboxes_labels (optional)

• gt_masks (optional)

• gt_ignore_flags (optional)

Parameters

• min_gt_bbox_wh (tuple[float]) – Minimum width and height of ground truth boxes. Default: (1., 1.)

• min_gt_mask_area (int) – Minimum foreground area of ground truth masks. Default: 1

• by_box (bool) – Filter instances with bounding boxes not meeting the min_gt_bbox_wh threshold. Default: True

• by_mask (bool) – Filter instances with masks not meeting min_gt_mask_area threshold. Default: False

• keep_empty (bool) – Whether to return None when it becomes an empty bbox after filtering. Defaults to True.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.FixScaleResize(scale: Optional[Union[int, Tuple[int, int]]] = None, scale_factor: Optional[Union[float, Tuple[float, float]]] = None, keep_ratio: bool = False, clip_object_border: bool = True, backend: str = 'cv2', interpolation='bilinear')

Compared to Resize, FixScaleResize fixes the scaling issue when keep_ratio=true.

class mmdet.datasets.transforms.FixShapeResize(width: int, height: int, pad_val: Union[int, float, dict] = {'img': 0, 'seg': 255}, keep_ratio: bool = False, clip_object_border: bool = True, backend: str = 'cv2', interpolation: str = 'bilinear')

Resize images & bbox & seg to the specified size.

This transform resizes the input image according to width and height. Bboxes, masks, and seg map are then resized with the same parameters.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• img_shape

• gt_bboxes

• gt_masks

• gt_seg_map

Added Keys:

• scale

• scale_factor

• keep_ratio

• homography_matrix

Parameters

• width (int) – width for resizing.

• height (int) – height for resizing. Defaults to None.

• pad_val (Number | dict[str, Number], optional) –

  Padding value for if the pad_mode is “constant”. If it is a single number, the value to pad the image is the number and to pad the semantic segmentation map is 255. If it is a dict, it should have the following keys:

  • img: The value to pad the image.

  • seg: The value to pad the semantic segmentation map.

  Defaults to dict(img=0, seg=255).

• keep_ratio (bool) – Whether to keep the aspect ratio when resizing the image. Defaults to False.

• clip_object_border (bool) – Whether to clip the objects outside the border of the image. In some dataset like MOT17, the gt bboxes are allowed to cross the border of images. Therefore, we don’t need to clip the gt bboxes in these cases. Defaults to True.

• backend (str) – Image resize backend, choices are ‘cv2’ and ‘pillow’. These two backends generates slightly different results. Defaults to ‘cv2’.

• interpolation (str) – Interpolation method, accepted values are “nearest”, “bilinear”, “bicubic”, “area”, “lanczos” for ‘cv2’ backend, “nearest”, “bilinear” for ‘pillow’ backend. Defaults to ‘bilinear’.

transform(results: dict, *args, **kwargs) → dict

Transform function to resize images, bounding boxes, semantic segmentation map and keypoints.

Parameters

results (dict) – Result dict from loading pipeline.

Returns

Resized results, ‘img’, ‘gt_bboxes’, ‘gt_seg_map’, ‘gt_keypoints’, ‘scale’, ‘scale_factor’, ‘img_shape’, and ‘keep_ratio’ keys are updated in result dict.

Return type

dict

class mmdet.datasets.transforms.GTBoxSubOne_GLIP

Subtract 1 from the x2 and y2 coordinates of the gt_bboxes.

transform(results: dict) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.GeomTransform(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.0, max_mag: float = 1.0, reversal_prob: float = 0.5, img_border_value: Union[int, float, tuple] = 128, mask_border_value: int = 0, seg_ignore_label: int = 255, interpolation: str = 'bilinear')

Base class for geometric transformations. All geometric transformations need to inherit from this base class. GeomTransform unifies the class attributes and class functions of geometric transformations (ShearX, ShearY, Rotate, TranslateX, and TranslateY), and records the homography matrix.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• gt_bboxes

• gt_masks

• gt_seg_map

Added Keys:

• homography_matrix

Parameters

• prob (float) – The probability for performing the geometric transformation and should be in range [0, 1]. Defaults to 1.0.

• level (int, optional) – The level should be in range [0, _MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum magnitude for geometric transformation. Defaults to 0.0.

• max_mag (float) – The maximum magnitude for geometric transformation. Defaults to 1.0.

• reversal_prob (float) – The probability that reverses the geometric transformation magnitude. Should be in range [0,1]. Defaults to 0.5.

• img_border_value (int | float | tuple) – The filled values for image border. If float, the same fill value will be used for all the three channels of image. If tuple, it should be 3 elements. Defaults to 128.

• mask_border_value (int) – The fill value used for masks. Defaults to 0.

• seg_ignore_label (int) – The fill value used for segmentation map. Note this value must equals ignore_label in semantic_head of the corresponding config. Defaults to 255.

• interpolation (str) – Interpolation method, accepted values are “nearest”, “bilinear”, “bicubic”, “area”, “lanczos” for ‘cv2’ backend, “nearest”, “bilinear” for ‘pillow’ backend. Defaults to ‘bilinear’.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.ImageToTensor(keys)

Convert image to torch.Tensor by given keys.

The dimension order of input image is (H, W, C). The pipeline will convert it to (C, H, W). If only 2 dimension (H, W) is given, the output would be (1, H, W).

Parameters

keys (Sequence[str]) – Key of images to be converted to Tensor.

class mmdet.datasets.transforms.InferencerLoader(**kwargs)

Load an image from results['img'].

Similar with LoadImageFromFile, but the image has been loaded as np.ndarray in results['img']. Can be used when loading image from webcam.

Required Keys:

• img

Modified Keys:

• img

• img_path

• img_shape

• ori_shape

Parameters

to_float32 (bool) – Whether to convert the loaded image to a float32 numpy array. If set to False, the loaded image is an uint8 array. Defaults to False.

transform(results: Union[str, ndarray, dict]) → dict

Transform function to add image meta information.

Parameters

results (str, np.ndarray or dict) – The result.

Returns

The dict contains loaded image and meta information.

Return type

dict

class mmdet.datasets.transforms.InstaBoost(action_candidate: tuple = ('normal', 'horizontal', 'skip'), action_prob: tuple = (1, 0, 0), scale: tuple = (0.8, 1.2), dx: int = 15, dy: int = 15, theta: tuple = (-1, 1), color_prob: float = 0.5, hflag: bool = False, aug_ratio: float = 0.5)

Data augmentation method in InstaBoost: Boosting Instance Segmentation Via Probability Map Guided Copy-Pasting.

Refer to https://github.com/GothicAi/Instaboost for implementation details.

Required Keys:

• img (np.uint8)

• instances

Modified Keys:

• img (np.uint8)

• instances

Parameters

• action_candidate (tuple) – Action candidates. “normal”, “horizontal”, “vertical”, “skip” are supported. Defaults to (‘normal’, ‘horizontal’, ‘skip’).

• action_prob (tuple) – Corresponding action probabilities. Should be the same length as action_candidate. Defaults to (1, 0, 0).

• scale (tuple) – (min scale, max scale). Defaults to (0.8, 1.2).

• dx (int) – The maximum x-axis shift will be (instance width) / dx. Defaults to 15.

• dy (int) – The maximum y-axis shift will be (instance height) / dy. Defaults to 15.

• theta (tuple) – (min rotation degree, max rotation degree). Defaults to (-1, 1).

• color_prob (float) – Probability of images for color augmentation. Defaults to 0.5.

• hflag (bool) – Whether to use heatmap guided. Defaults to False.

• aug_ratio (float) – Probability of applying this transformation. Defaults to 0.5.

transform(results) → dict

The transform function.

class mmdet.datasets.transforms.Invert(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.1, max_mag: float = 1.9)

Invert images.

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing invert therefore should be in range [0, 1]. Defaults to 1.0.

• level (int, optional) – No use for Invert transformation. Defaults to None.

• min_mag (float) – No use for Invert transformation. Defaults to 0.1.

• max_mag (float) – No use for Invert transformation. Defaults to 1.9.

class mmdet.datasets.transforms.LineDetDataProcessor(mean: Optional[Sequence[Number]] = None, std: Optional[Sequence[Number]] = None, pad_size_divisor: int = 1, pad_value: Union[float, int] = 0, pad_mask: bool = False, mask_pad_value: int = 0, pad_seg: bool = True, seg_pad_value: int = 255, bgr_to_rgb: bool = False, rgb_to_bgr: bool = False, boxtype2tensor: bool = True, non_blocking: Optional[bool] = False, batch_augments: Optional[List[dict]] = None)

Image pre-processor for detection tasks.

Comparing with the mmengine.ImgDataPreprocessor,

1. It supports batch augmentations.

2. It will additionally append batch_input_shape and pad_shape to data_samples considering the object detection task.

It provides the data pre-processing as follows

• Collate and move data to the target device.

• Pad inputs to the maximum size of current batch with defined pad_value. The padding size can be divisible by a defined pad_size_divisor

• Stack inputs to batch_inputs.

• Convert inputs from bgr to rgb if the shape of input is (3, H, W).

• Normalize image with defined std and mean.

• Do batch augmentations during training.

Parameters

• mean (Sequence[Number], optional) – The pixel mean of R, G, B channels. Defaults to None.

• std (Sequence[Number], optional) – The pixel standard deviation of R, G, B channels. Defaults to None.

• pad_size_divisor (int) – The size of padded image should be divisible by pad_size_divisor. Defaults to 1.

• pad_value (Number) – The padded pixel value. Defaults to 0.

• pad_mask (bool) – Whether to pad instance masks. Defaults to False.

• mask_pad_value (int) – The padded pixel value for instance masks. Defaults to 0.

• pad_seg (bool) – Whether to pad semantic segmentation maps. Defaults to False.

• seg_pad_value (int) – The padded pixel value for semantic segmentation maps. Defaults to 255.

• bgr_to_rgb (bool) – whether to convert image from BGR to RGB. Defaults to False.

• rgb_to_bgr (bool) – whether to convert image from RGB to RGB. Defaults to False.

• boxtype2tensor (bool) – Whether to convert the BaseBoxes type of bboxes data to Tensor type. Defaults to True.

• non_blocking (bool) – Whether block current process when transferring data to device. Defaults to False.

• batch_augments (list[dict], optional) – Batch-level augmentations

forward(data: dict, training: bool = False) → dict

Perform normalization,padding and bgr2rgb conversion based on BaseDataPreprocessor.

Parameters

• data (dict) – Data sampled from dataloader.

• training (bool) – Whether to enable training time augmentation.

Returns

Data in the same format as the model input.

Return type

dict

pad_gt_masks(batch_data_samples: Sequence[DetDataSample]) → None

Pad gt_masks to shape of batch_input_shape.

pad_gt_sem_seg(batch_data_samples: Sequence[DetDataSample]) → None

Pad gt_sem_seg to shape of batch_input_shape.

class mmdet.datasets.transforms.LinesToArray(num_points: int = 72, max_lines: int = 4, img_height: int = 320, img_width: int = 800)

transform(result)

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.LoadAnnotations(with_mask: bool = False, poly2mask: bool = True, box_type: str = 'hbox', reduce_zero_label: bool = False, ignore_index: int = 255, **kwargs)

Load and process the instances and seg_map annotation provided by dataset.

The annotation format is as the following:

{
    'instances':
    [
        {
        # List of 4 numbers representing the bounding box of the
        # instance, in (x1, y1, x2, y2) order.
        'bbox': [x1, y1, x2, y2],

        # Label of image classification.
        'bbox_label': 1,

        # Used in instance/panoptic segmentation. The segmentation mask
        # of the instance or the information of segments.
        # 1. If list[list[float]], it represents a list of polygons,
        # one for each connected component of the object. Each
        # list[float] is one simple polygon in the format of
        # [x1, y1, ..., xn, yn] (n >= 3). The Xs and Ys are absolute
        # coordinates in unit of pixels.
        # 2. If dict, it represents the per-pixel segmentation mask in
        # COCO's compressed RLE format. The dict should have keys
        # “size” and “counts”.  Can be loaded by pycocotools
        'mask': list[list[float]] or dict,

        }
    ]
    # Filename of semantic or panoptic segmentation ground truth file.
    'seg_map_path': 'a/b/c'
}

After this module, the annotation has been changed to the format below:

{
    # In (x1, y1, x2, y2) order, float type. N is the number of bboxes
    # in an image
    'gt_bboxes': BaseBoxes(N, 4)
     # In int type.
    'gt_bboxes_labels': np.ndarray(N, )
     # In built-in class
    'gt_masks': PolygonMasks (H, W) or BitmapMasks (H, W)
     # In uint8 type.
    'gt_seg_map': np.ndarray (H, W)
     # in (x, y, v) order, float type.
}

Required Keys:

• height

• width

• instances

  • bbox (optional)

  • bbox_label

  • mask (optional)

  • ignore_flag

• seg_map_path (optional)

Added Keys:

• gt_bboxes (BaseBoxes[torch.float32])

• gt_bboxes_labels (np.int64)

• gt_masks (BitmapMasks | PolygonMasks)

• gt_seg_map (np.uint8)

• gt_ignore_flags (bool)

Parameters

• with_bbox (bool) – Whether to parse and load the bbox annotation. Defaults to True.

• with_label (bool) – Whether to parse and load the label annotation. Defaults to True.

• with_mask (bool) – Whether to parse and load the mask annotation. Default: False.

• with_seg (bool) – Whether to parse and load the semantic segmentation annotation. Defaults to False.

• poly2mask (bool) – Whether to convert mask to bitmap. Default: True.

• box_type (str) – The box type used to wrap the bboxes. If box_type is None, gt_bboxes will keep being np.ndarray. Defaults to ‘hbox’.

• reduce_zero_label (bool) – Whether reduce all label value by 1. Usually used for datasets where 0 is background label. Defaults to False.

• ignore_index (int) – The label index to be ignored. Valid only if reduce_zero_label is true. Defaults is 255.

• imdecode_backend (str) – The image decoding backend type. The backend argument for :func:mmcv.imfrombytes. See :fun:mmcv.imfrombytes for details. Defaults to ‘cv2’.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

transform(results: dict) → dict

Function to load multiple types annotations.

Parameters

results (dict) – Result dict from :obj:mmengine.BaseDataset.

Returns

The dict contains loaded bounding box, label and semantic segmentation.

Return type

dict

class mmdet.datasets.transforms.LoadEmptyAnnotations(with_bbox: bool = True, with_label: bool = True, with_mask: bool = False, with_seg: bool = False, seg_ignore_label: int = 255)

Load Empty Annotations for unlabeled images.

Added Keys: - gt_bboxes (np.float32) - gt_bboxes_labels (np.int64) - gt_masks (BitmapMasks | PolygonMasks) - gt_seg_map (np.uint8) - gt_ignore_flags (bool)

Parameters

• with_bbox (bool) – Whether to load the pseudo bbox annotation. Defaults to True.

• with_label (bool) – Whether to load the pseudo label annotation. Defaults to True.

• with_mask (bool) – Whether to load the pseudo mask annotation. Default: False.

• with_seg (bool) – Whether to load the pseudo semantic segmentation annotation. Defaults to False.

• seg_ignore_label (int) – The fill value used for segmentation map. Note this value must equals ignore_label in semantic_head of the corresponding config. Defaults to 255.

transform(results: dict) → dict

Transform function to load empty annotations.

Parameters

results (dict) – Result dict.

Returns

Updated result dict.

Return type

dict

class mmdet.datasets.transforms.LoadImageFromNDArray(to_float32: bool = False, color_type: str = 'color', imdecode_backend: str = 'cv2', file_client_args: Optional[dict] = None, ignore_empty: bool = False, *, backend_args: Optional[dict] = None)

Load an image from results['img'].

Similar with LoadImageFromFile, but the image has been loaded as np.ndarray in results['img']. Can be used when loading image from webcam.

Required Keys:

• img

Modified Keys:

• img

• img_path

• img_shape

• ori_shape

Parameters

to_float32 (bool) – Whether to convert the loaded image to a float32 numpy array. If set to False, the loaded image is an uint8 array. Defaults to False.

transform(results: dict) → dict

Transform function to add image meta information.

Parameters

results (dict) – Result dict with Webcam read image in results['img'].

Returns

The dict contains loaded image and meta information.

Return type

dict

class mmdet.datasets.transforms.LoadMultiChannelImageFromFiles(to_float32: bool = False, color_type: str = 'unchanged', imdecode_backend: str = 'cv2', file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None)

Load multi-channel images from a list of separate channel files.

Required Keys:

• img_path

Modified Keys:

• img

• img_shape

• ori_shape

Parameters

• to_float32 (bool) – Whether to convert the loaded image to a float32 numpy array. If set to False, the loaded image is an uint8 array. Defaults to False.

• color_type (str) – The flag argument for :func:mmcv.imfrombytes. Defaults to ‘unchanged’.

• imdecode_backend (str) – The image decoding backend type. The backend argument for :func:mmcv.imfrombytes. See :func:mmcv.imfrombytes for details. Defaults to ‘cv2’.

• file_client_args (dict) – Arguments to instantiate the corresponding backend in mmdet <= 3.0.0rc6. Defaults to None.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend in mmdet >= 3.0.0rc7. Defaults to None.

transform(results: dict) → dict

Transform functions to load multiple images and get images meta information.

Parameters

results (dict) – Result dict from mmdet.CustomDataset.

Returns

The dict contains loaded images and meta information.

Return type

dict

class mmdet.datasets.transforms.LoadPanopticAnnotations(with_bbox: bool = True, with_label: bool = True, with_mask: bool = True, with_seg: bool = True, box_type: str = 'hbox', imdecode_backend: str = 'cv2', backend_args: Optional[dict] = None)

Load multiple types of panoptic annotations.

The annotation format is as the following:

{
    'instances':
    [
        {
        # List of 4 numbers representing the bounding box of the
        # instance, in (x1, y1, x2, y2) order.
        'bbox': [x1, y1, x2, y2],

        # Label of image classification.
        'bbox_label': 1,
        },
        ...
    ]
    'segments_info':
    [
        {
        # id = cls_id + instance_id * INSTANCE_OFFSET
        'id': int,

        # Contiguous category id defined in dataset.
        'category': int

        # Thing flag.
        'is_thing': bool
        },
        ...
    ]

    # Filename of semantic or panoptic segmentation ground truth file.
    'seg_map_path': 'a/b/c'
}

After this module, the annotation has been changed to the format below:

{
    # In (x1, y1, x2, y2) order, float type. N is the number of bboxes
    # in an image
    'gt_bboxes': BaseBoxes(N, 4)
     # In int type.
    'gt_bboxes_labels': np.ndarray(N, )
     # In built-in class
    'gt_masks': PolygonMasks (H, W) or BitmapMasks (H, W)
     # In uint8 type.
    'gt_seg_map': np.ndarray (H, W)
     # in (x, y, v) order, float type.
}

Required Keys:

• height

• width

• instances - bbox - bbox_label - ignore_flag

• segments_info - id - category - is_thing

• seg_map_path

Added Keys:

• gt_bboxes (BaseBoxes[torch.float32])

• gt_bboxes_labels (np.int64)

• gt_masks (BitmapMasks | PolygonMasks)

• gt_seg_map (np.uint8)

• gt_ignore_flags (bool)

Parameters

• with_bbox (bool) – Whether to parse and load the bbox annotation. Defaults to True.

• with_label (bool) – Whether to parse and load the label annotation. Defaults to True.

• with_mask (bool) – Whether to parse and load the mask annotation. Defaults to True.

• with_seg (bool) – Whether to parse and load the semantic segmentation annotation. Defaults to False.

• box_type (str) – The box mode used to wrap the bboxes.

• imdecode_backend (str) – The image decoding backend type. The backend argument for :func:mmcv.imfrombytes. See :fun:mmcv.imfrombytes for details. Defaults to ‘cv2’.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend in mmdet >= 3.0.0rc7. Defaults to None.

transform(results: dict) → dict

Function to load multiple types panoptic annotations.

Parameters

results (dict) – Result dict from :obj:mmdet.CustomDataset.

Returns

The dict contains loaded bounding box, label, mask and

semantic segmentation annotations.

Return type

dict

class mmdet.datasets.transforms.LoadProposals(num_max_proposals: Optional[int] = None)

Load proposal pipeline.

Required Keys:

• proposals

Modified Keys:

• proposals

Parameters

num_max_proposals (int, optional) – Maximum number of proposals to load. If not specified, all proposals will be loaded.

transform(results: dict) → dict

Transform function to load proposals from file.

Parameters

results (dict) – Result dict from mmdet.CustomDataset.

Returns

The dict contains loaded proposal annotations.

Return type

dict

class mmdet.datasets.transforms.LoadTextAnnotations

transform(results: dict) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.LoadTrackAnnotations(**kwargs)

Load and process the instances and seg_map annotation provided by dataset. It must load instances_ids which is only used in the tracking tasks. The annotation format is as the following:

After this module, the annotation has been changed to the format below: .. code-block:: python

{

# In (x1, y1, x2, y2) order, float type. N is the number of bboxes # in an image ‘gt_bboxes’: np.ndarray(N, 4)

# In int type.

‘gt_bboxes_labels’: np.ndarray(N, )

# In built-in class

‘gt_masks’: PolygonMasks (H, W) or BitmapMasks (H, W)

# In uint8 type.

‘gt_seg_map’: np.ndarray (H, W)

# in (x, y, v) order, float type.

}

Required Keys:

• height (optional)

• width (optional)

• instances - bbox (optional) - bbox_label - instance_id (optional) - mask (optional) - ignore_flag (optional)

• seg_map_path (optional)

Added Keys:

• gt_bboxes (np.float32)

• gt_bboxes_labels (np.int32)

• gt_instances_ids (np.int32)

• gt_masks (BitmapMasks | PolygonMasks)

• gt_seg_map (np.uint8)

• gt_ignore_flags (np.bool_)

transform(results: dict) → dict

Function to load multiple types annotations.

Parameters

results (dict) – Result dict from :obj:mmcv.BaseDataset.

Returns

The dict contains loaded bounding box, label, instances id and semantic segmentation and keypoints annotations.

Return type

dict

class mmdet.datasets.transforms.MinIoURandomCrop(min_ious: Sequence[float] = (0.1, 0.3, 0.5, 0.7, 0.9), min_crop_size: float = 0.3, bbox_clip_border: bool = True)

Random crop the image & bboxes & masks & segmentation map, the cropped patches have minimum IoU requirement with original image & bboxes & masks.

& segmentation map, the IoU threshold is randomly selected from min_ious.

Required Keys:

• img

• img_shape

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_ignore_flags (bool) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• img_shape

• gt_bboxes

• gt_bboxes_labels

• gt_masks

• gt_ignore_flags

• gt_seg_map

Parameters

• min_ious (Sequence[float]) – minimum IoU threshold for all intersections with bounding boxes.

• min_crop_size (float) – minimum crop’s size (i.e. h,w := a*h, a*w,

• min_crop_size). (where a >=) –

• bbox_clip_border (bool, optional) – Whether clip the objects outside the border of the image. Defaults to True.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.MixUp(img_scale: Tuple[int, int] = (640, 640), ratio_range: Tuple[float, float] = (0.5, 1.5), flip_ratio: float = 0.5, pad_val: float = 114.0, max_iters: int = 15, bbox_clip_border: bool = True)

MixUp data augmentation.

                    mixup transform
           +------------------------------+
           | mixup image   |              |
           |      +--------|--------+     |
           |      |        |        |     |
           |---------------+        |     |
           |      |                 |     |
           |      |      image      |     |
           |      |                 |     |
           |      |                 |     |
           |      |-----------------+     |
           |             pad              |
           +------------------------------+

The mixup transform steps are as follows:

   1. Another random image is picked by dataset and embedded in
      the top left patch(after padding and resizing)
   2. The target of mixup transform is the weighted average of mixup
      image and origin image.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_ignore_flags (bool) (optional)

• mix_results (List[dict])

Modified Keys:

• img

• img_shape

• gt_bboxes (optional)

• gt_bboxes_labels (optional)

• gt_ignore_flags (optional)

Parameters

• img_scale (Sequence[int]) – Image output size after mixup pipeline. The shape order should be (width, height). Defaults to (640, 640).

• ratio_range (Sequence[float]) – Scale ratio of mixup image. Defaults to (0.5, 1.5).

• flip_ratio (float) – Horizontal flip ratio of mixup image. Defaults to 0.5.

• pad_val (int) – Pad value. Defaults to 114.

• max_iters (int) – The maximum number of iterations. If the number of iterations is greater than max_iters, but gt_bbox is still empty, then the iteration is terminated. Defaults to 15.

• bbox_clip_border (bool, optional) – Whether to clip the objects outside the border of the image. In some dataset like MOT17, the gt bboxes are allowed to cross the border of images. Therefore, we don’t need to clip the gt bboxes in these cases. Defaults to True.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.Mosaic(img_scale: Tuple[int, int] = (640, 640), center_ratio_range: Tuple[float, float] = (0.5, 1.5), bbox_clip_border: bool = True, pad_val: float = 114.0, prob: float = 1.0)

Mosaic augmentation.

Given 4 images, mosaic transform combines them into one output image. The output image is composed of the parts from each sub- image.

                   mosaic transform
                      center_x
           +------------------------------+
           |       pad        |  pad      |
           |      +-----------+           |
           |      |           |           |
           |      |  image1   |--------+  |
           |      |           |        |  |
           |      |           | image2 |  |
center_y   |----+-------------+-----------|
           |    |   cropped   |           |
           |pad |   image3    |  image4   |
           |    |             |           |
           +----|-------------+-----------+
                |             |
                +-------------+

The mosaic transform steps are as follows:

    1. Choose the mosaic center as the intersections of 4 images
    2. Get the left top image according to the index, and randomly
       sample another 3 images from the custom dataset.
    3. Sub image will be cropped if image is larger than mosaic patch

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_ignore_flags (bool) (optional)

• mix_results (List[dict])

Modified Keys:

• img

• img_shape

• gt_bboxes (optional)

• gt_bboxes_labels (optional)

• gt_ignore_flags (optional)

Parameters

• img_scale (Sequence[int]) – Image size before mosaic pipeline of single image. The shape order should be (width, height). Defaults to (640, 640).

• center_ratio_range (Sequence[float]) – Center ratio range of mosaic output. Defaults to (0.5, 1.5).

• bbox_clip_border (bool, optional) – Whether to clip the objects outside the border of the image. In some dataset like MOT17, the gt bboxes are allowed to cross the border of images. Therefore, we don’t need to clip the gt bboxes in these cases. Defaults to True.

• pad_val (int) – Pad value. Defaults to 114.

• prob (float) – Probability of applying this transformation. Defaults to 1.0.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.MultiBranch(branch_field: List[str], **branch_pipelines: dict)

Multiple branch pipeline wrapper.

Generate multiple data-augmented versions of the same image. MultiBranch needs to specify the branch names of all pipelines of the dataset, perform corresponding data augmentation for the current branch, and return None for other branches, which ensures the consistency of return format across different samples.

Parameters

• branch_field (list) – List of branch names.

• branch_pipelines (dict) – Dict of different pipeline configs to be composed.

Examples

>>> branch_field = ['sup', 'unsup_teacher', 'unsup_student']
>>> sup_pipeline = [
>>>     dict(type='LoadImageFromFile'),
>>>     dict(type='LoadAnnotations', with_bbox=True),
>>>     dict(type='Resize', scale=(1333, 800), keep_ratio=True),
>>>     dict(type='RandomFlip', prob=0.5),
>>>     dict(
>>>         type='MultiBranch',
>>>         branch_field=branch_field,
>>>         sup=dict(type='PackDetInputs'))
>>>     ]
>>> weak_pipeline = [
>>>     dict(type='LoadImageFromFile'),
>>>     dict(type='LoadAnnotations', with_bbox=True),
>>>     dict(type='Resize', scale=(1333, 800), keep_ratio=True),
>>>     dict(type='RandomFlip', prob=0.0),
>>>     dict(
>>>         type='MultiBranch',
>>>         branch_field=branch_field,
>>>         sup=dict(type='PackDetInputs'))
>>>     ]
>>> strong_pipeline = [
>>>     dict(type='LoadImageFromFile'),
>>>     dict(type='LoadAnnotations', with_bbox=True),
>>>     dict(type='Resize', scale=(1333, 800), keep_ratio=True),
>>>     dict(type='RandomFlip', prob=1.0),
>>>     dict(
>>>         type='MultiBranch',
>>>         branch_field=branch_field,
>>>         sup=dict(type='PackDetInputs'))
>>>     ]
>>> unsup_pipeline = [
>>>     dict(type='LoadImageFromFile'),
>>>     dict(type='LoadEmptyAnnotations'),
>>>     dict(
>>>         type='MultiBranch',
>>>         branch_field=branch_field,
>>>         unsup_teacher=weak_pipeline,
>>>         unsup_student=strong_pipeline)
>>>     ]
>>> from mmcv.transforms import Compose
>>> sup_branch = Compose(sup_pipeline)
>>> unsup_branch = Compose(unsup_pipeline)
>>> print(sup_branch)
>>> Compose(
>>>     LoadImageFromFile(ignore_empty=False, to_float32=False, color_type='color', imdecode_backend='cv2') # noqa
>>>     LoadAnnotations(with_bbox=True, with_label=True, with_mask=False, with_seg=False, poly2mask=True, imdecode_backend='cv2') # noqa
>>>     Resize(scale=(1333, 800), scale_factor=None, keep_ratio=True, clip_object_border=True), backend=cv2), interpolation=bilinear) # noqa
>>>     RandomFlip(prob=0.5, direction=horizontal)
>>>     MultiBranch(branch_pipelines=['sup'])
>>> )
>>> print(unsup_branch)
>>> Compose(
>>>     LoadImageFromFile(ignore_empty=False, to_float32=False, color_type='color', imdecode_backend='cv2') # noqa
>>>     LoadEmptyAnnotations(with_bbox=True, with_label=True, with_mask=False, with_seg=False, seg_ignore_label=255) # noqa
>>>     MultiBranch(branch_pipelines=['unsup_teacher', 'unsup_student'])
>>> )

transform(results: dict) → dict

Transform function to apply transforms sequentially.

Parameters

results (dict) – Result dict from loading pipeline.

Returns

• ‘inputs’ (Dict[str, obj:torch.Tensor]): The forward data of

  models from different branches.

• ’data_sample’ (Dict[str,obj:DetDataSample]): The annotation

  info of the sample from different branches.

Return type

dict

class mmdet.datasets.transforms.PackDetInputs(meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', 'flip', 'flip_direction'))

Pack the inputs data for the detection / semantic segmentation / panoptic segmentation.

The img_meta item is always populated. The contents of the img_meta dictionary depends on meta_keys. By default this includes:

• img_id: id of the image

• img_path: path to the image file

• ori_shape: original shape of the image as a tuple (h, w)

• img_shape: shape of the image input to the network as a tuple (h, w). Note that images may be zero padded on the bottom/right if the batch tensor is larger than this shape.

• scale_factor: a float indicating the preprocessing scale

• flip: a boolean indicating if image flip transform was used

• flip_direction: the flipping direction

Parameters

meta_keys (Sequence[str], optional) – Meta keys to be converted to mmcv.DataContainer and collected in data[img_metas]. Default: ('img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', 'flip', 'flip_direction')

transform(results: dict) → dict

Method to pack the input data.

Parameters

results (dict) – Result dict from the data pipeline.

Returns

• ‘inputs’ (obj:torch.Tensor): The forward data of models.

• ’data_sample’ (obj:DetDataSample): The annotation info of the

  sample.

Return type

dict

class mmdet.datasets.transforms.PackLineDetectionInputs

transform(results: Dict) → Optional[Union[Dict, Tuple[List, List]]]

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.PackReIDInputs(meta_keys: Sequence[str] = ())

Pack the inputs data for the ReID. The meta_info item is always populated. The contents of the meta_info dictionary depends on meta_keys. By default this includes:

• img_path: path to the image file.

• ori_shape: original shape of the image as a tuple (H, W).

• img_shape: shape of the image input to the network as a tuple

  (H, W). Note that images may be zero padded on the bottom/right

  if the batch tensor is larger than this shape.

• scale: scale of the image as a tuple (W, H).

• scale_factor: a float indicating the pre-processing scale.

• flip: a boolean indicating if image flip transform was used.

• flip_direction: the flipping direction.

Parameters

meta_keys (Sequence[str], optional) – The meta keys to saved in the metainfo of the packed data_sample.

transform(results: dict) → dict

Method to pack the input data. :param results: Result dict from the data pipeline. :type results: dict

Returns

• ‘inputs’ (dict[Tensor]): The forward data of models.

• ’data_samples’ (obj:ReIDDataSample): The meta info of the

  sample.

Return type

dict

class mmdet.datasets.transforms.PackTrackInputs(meta_keys: Optional[dict] = None, default_meta_keys: tuple = ('img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', 'flip', 'flip_direction', 'frame_id', 'video_id', 'video_length', 'ori_video_length', 'instances'))

Pack the inputs data for the multi object tracking and video instance segmentation. All the information of images are packed to inputs. All the information except images are packed to data_samples. In order to get the original annotaiton and meta info, we add instances key into meta keys.

Parameters

• meta_keys (Sequence[str]) – Meta keys to be collected in data_sample.metainfo. Defaults to None.

• default_meta_keys (tuple) – Default meta keys. Defaults to (‘img_id’, ‘img_path’, ‘ori_shape’, ‘img_shape’, ‘scale_factor’, ‘flip’, ‘flip_direction’, ‘frame_id’, ‘is_video_data’, ‘video_id’, ‘video_length’, ‘instances’).

transform(results: dict) → dict

Method to pack the input data. :param results: Result dict from the data pipeline. :type results: dict

Returns

• ‘inputs’ (dict[Tensor]): The forward data of models.

• ’data_samples’ (obj:TrackDataSample): The annotation info of

  the samples.

Return type

dict

class mmdet.datasets.transforms.Pad(size: Optional[Tuple[int, int]] = None, size_divisor: Optional[int] = None, pad_to_square: bool = False, pad_val: Union[int, float, dict] = {'img': 0, 'seg': 255}, padding_mode: str = 'constant')

Pad the image & segmentation map.

There are three padding modes: (1) pad to a fixed size and (2) pad to the minimum size that is divisible by some number. and (3)pad to square. Also, pad to square and pad to the minimum size can be used as the same time.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• img_shape

• gt_masks

• gt_seg_map

Added Keys:

• pad_shape

• pad_fixed_size

• pad_size_divisor

Parameters

• size (tuple, optional) – Fixed padding size. Expected padding shape (width, height). Defaults to None.

• size_divisor (int, optional) – The divisor of padded size. Defaults to None.

• pad_to_square (bool) – Whether to pad the image into a square. Currently only used for YOLOX. Defaults to False.

• pad_val (Number | dict[str, Number], optional) –

  the pad_mode is “constant”. If it is a single number, the value to pad the image is the number and to pad the semantic segmentation map is 255. If it is a dict, it should have the following keys:

  • img: The value to pad the image.

  • seg: The value to pad the semantic segmentation map.

  Defaults to dict(img=0, seg=255).

• padding_mode (str) –

  Type of padding. Should be: constant, edge, reflect or symmetric. Defaults to ‘constant’.

  • constant: pads with a constant value, this value is specified with pad_val.

  • edge: pads with the last value at the edge of the image.

  • reflect: pads with reflection of image without repeating the last value on the edge. For example, padding [1, 2, 3, 4] with 2 elements on both sides in reflect mode will result in [3, 2, 1, 2, 3, 4, 3, 2].

  • symmetric: pads with reflection of image repeating the last value on the edge. For example, padding [1, 2, 3, 4] with 2 elements on both sides in symmetric mode will result in [2, 1, 1, 2, 3, 4, 4, 3]

transform(results: dict) → dict

Call function to pad images, masks, semantic segmentation maps.

Parameters

results (dict) – Result dict from loading pipeline.

Returns

Updated result dict.

Return type

dict

class mmdet.datasets.transforms.PhotoMetricDistortion(brightness_delta: int = 32, contrast_range: Sequence[Union[float, int]] = (0.5, 1.5), saturation_range: Sequence[Union[float, int]] = (0.5, 1.5), hue_delta: int = 18)

Apply photometric distortion to image sequentially, every transformation is applied with a probability of 0.5. The position of random contrast is in second or second to last.

1. random brightness

2. random contrast (mode 0)

3. convert color from BGR to HSV

4. random saturation

5. random hue

6. convert color from HSV to BGR

7. random contrast (mode 1)

8. randomly swap channels

Required Keys:

• img (np.uint8)

Modified Keys:

• img (np.float32)

Parameters

• brightness_delta (int) – delta of brightness.

• contrast_range (sequence) – range of contrast.

• saturation_range (sequence) – range of saturation.

• hue_delta (int) – delta of hue.

transform(results: dict) → dict

Transform function to perform photometric distortion on images.

Parameters

results (dict) – Result dict from loading pipeline.

Returns

Result dict with images distorted.

Return type

dict

class mmdet.datasets.transforms.Posterize(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.0, max_mag: float = 4.0)

Posterize images (reduce the number of bits for each color channel).

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing Posterize transformation. Defaults to 1.0.

• level (int, optional) – Should be in range [0,_MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum magnitude for Posterize transformation. Defaults to 0.0.

• max_mag (float) – The maximum magnitude for Posterize transformation. Defaults to 4.0.

class mmdet.datasets.transforms.ProposalBroadcaster(transforms: List[Union[dict, Callable]] = [])

A transform wrapper to apply the wrapped transforms to process both gt_bboxes and proposals without adding any codes. It will do the following steps:

1. Scatter the broadcasting targets to a list of inputs of the wrapped transforms. The type of the list should be list[dict, dict], which the first is the original inputs, the second is the processing results that gt_bboxes being rewritten by the proposals.

2. Apply self.transforms, with same random parameters, which is sharing with a context manager. The type of the outputs is a list[dict, dict].

3. Gather the outputs, update the proposals in the first item of the outputs with the gt_bboxes in the second .

Parameters

transforms (list, optional) – Sequence of transform object or config dict to be wrapped. Defaults to [].

Note: The TransformBroadcaster in MMCV can achieve the same operation as

ProposalBroadcaster, but need to set more complex parameters.

Examples

>>> pipeline = [
>>>     dict(type='LoadImageFromFile'),
>>>     dict(type='LoadProposals', num_max_proposals=2000),
>>>     dict(type='LoadAnnotations', with_bbox=True),
>>>     dict(
>>>         type='ProposalBroadcaster',
>>>         transforms=[
>>>             dict(type='Resize', scale=(1333, 800),
>>>                  keep_ratio=True),
>>>             dict(type='RandomFlip', prob=0.5),
>>>         ]),
>>>     dict(type='PackDetInputs')]

transform(results: dict) → dict

Apply wrapped transform functions to process both gt_bboxes and proposals.

Parameters

results (dict) – Result dict from loading pipeline.

Returns

Updated result dict.

Return type

dict

class mmdet.datasets.transforms.RandAugment(aug_space: List[Union[ConfigDict, dict]] = [[{'type': 'AutoContrast'}], [{'type': 'Equalize'}], [{'type': 'Invert'}], [{'type': 'Rotate'}], [{'type': 'Posterize'}], [{'type': 'Solarize'}], [{'type': 'SolarizeAdd'}], [{'type': 'Color'}], [{'type': 'Contrast'}], [{'type': 'Brightness'}], [{'type': 'Sharpness'}], [{'type': 'ShearX'}], [{'type': 'ShearY'}], [{'type': 'TranslateX'}], [{'type': 'TranslateY'}]], aug_num: int = 2, prob: Optional[List[float]] = None)

Rand augmentation.

This data augmentation is proposed in RandAugment: Practical automated data augmentation with a reduced search space.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_ignore_flags (bool) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• img_shape

• gt_bboxes

• gt_bboxes_labels

• gt_masks

• gt_ignore_flags

• gt_seg_map

Added Keys:

• homography_matrix

Parameters

• aug_space (List[List[Union[dict, ConfigDict]]]) – The augmentation space of rand augmentation. Each augmentation transform in aug_space is a specific transform, and is composed by several augmentations. When RandAugment is called, a random transform in aug_space will be selected to augment images. Defaults to aug_space.

• aug_num (int) – Number of augmentation to apply equentially. Defaults to 2.

• prob (list[float], optional) – The probabilities associated with each augmentation. The length should be equal to the augmentation space and the sum should be 1. If not given, a uniform distribution will be assumed. Defaults to None.

Examples

>>> aug_space = [
>>>     dict(type='Sharpness'),
>>>     dict(type='ShearX'),
>>>     dict(type='Color'),
>>>     ],
>>> augmentation = RandAugment(aug_space)
>>> img = np.ones(100, 100, 3)
>>> gt_bboxes = np.ones(10, 4)
>>> results = dict(img=img, gt_bboxes=gt_bboxes)
>>> results = augmentation(results)

random_pipeline_index()

Return a random transform index.

transform(results: dict) → dict

Transform function to use RandAugment.

Parameters

results (dict) – Result dict from loading pipeline.

Returns

Result dict with RandAugment.

Return type

dict

class mmdet.datasets.transforms.RandomAffine(max_rotate_degree: float = 10.0, max_translate_ratio: float = 0.1, scaling_ratio_range: Tuple[float, float] = (0.5, 1.5), max_shear_degree: float = 2.0, border: Tuple[int, int] = (0, 0), border_val: Tuple[int, int, int] = (114, 114, 114), bbox_clip_border: bool = True)

Random affine transform data augmentation.

This operation randomly generates affine transform matrix which including rotation, translation, shear and scaling transforms.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_ignore_flags (bool) (optional)

Modified Keys:

• img

• img_shape

• gt_bboxes (optional)

• gt_bboxes_labels (optional)

• gt_ignore_flags (optional)

Parameters

• max_rotate_degree (float) – Maximum degrees of rotation transform. Defaults to 10.

• max_translate_ratio (float) – Maximum ratio of translation. Defaults to 0.1.

• scaling_ratio_range (tuple[float]) – Min and max ratio of scaling transform. Defaults to (0.5, 1.5).

• max_shear_degree (float) – Maximum degrees of shear transform. Defaults to 2.

• border (tuple[int]) – Distance from width and height sides of input image to adjust output shape. Only used in mosaic dataset. Defaults to (0, 0).

• border_val (tuple[int]) – Border padding values of 3 channels. Defaults to (114, 114, 114).

• bbox_clip_border (bool, optional) – Whether to clip the objects outside the border of the image. In some dataset like MOT17, the gt bboxes are allowed to cross the border of images. Therefore, we don’t need to clip the gt bboxes in these cases. Defaults to True.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.RandomCenterCropPad(crop_size: Optional[tuple] = None, ratios: Optional[tuple] = (0.9, 1.0, 1.1), border: Optional[int] = 128, mean: Optional[Sequence] = None, std: Optional[Sequence] = None, to_rgb: Optional[bool] = None, test_mode: bool = False, test_pad_mode: Optional[tuple] = ('logical_or', 127), test_pad_add_pix: int = 0, bbox_clip_border: bool = True)

Random center crop and random around padding for CornerNet.

This operation generates randomly cropped image from the original image and pads it simultaneously. Different from RandomCrop, the output shape may not equal to crop_size strictly. We choose a random value from ratios and the output shape could be larger or smaller than crop_size. The padding operation is also different from Pad, here we use around padding instead of right-bottom padding.

The relation between output image (padding image) and original image:

                output image

       +----------------------------+
       |          padded area       |
+------|----------------------------|----------+
|      |         cropped area       |          |
|      |         +---------------+  |          |
|      |         |    .   center |  |          | original image
|      |         |        range  |  |          |
|      |         +---------------+  |          |
+------|----------------------------|----------+
       |          padded area       |
       +----------------------------+

There are 5 main areas in the figure:

• output image: output image of this operation, also called padding image in following instruction.

• original image: input image of this operation.

• padded area: non-intersect area of output image and original image.

• cropped area: the overlap of output image and original image.

• center range: a smaller area where random center chosen from. center range is computed by border and original image’s shape to avoid our random center is too close to original image’s border.

Also this operation act differently in train and test mode, the summary pipeline is listed below.

Train pipeline:

1. Choose a random_ratio from ratios, the shape of padding image will be random_ratio * crop_size.

2. Choose a random_center in center range.

3. Generate padding image with center matches the random_center.

4. Initialize the padding image with pixel value equals to mean.

5. Copy the cropped area to padding image.

6. Refine annotations.

Test pipeline:

1. Compute output shape according to test_pad_mode.

2. Generate padding image with center matches the original image center.

3. Initialize the padding image with pixel value equals to mean.

4. Copy the cropped area to padding image.

Required Keys:

• img (np.float32)

• img_shape (tuple)

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_ignore_flags (bool) (optional)

Modified Keys:

• img (np.float32)

• img_shape (tuple)

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_ignore_flags (bool) (optional)

Parameters

• crop_size (tuple, optional) – expected size after crop, final size will computed according to ratio. Requires (width, height) in train mode, and None in test mode.

• ratios (tuple, optional) – random select a ratio from tuple and crop image to (crop_size[0] * ratio) * (crop_size[1] * ratio). Only available in train mode. Defaults to (0.9, 1.0, 1.1).

• border (int, optional) – max distance from center select area to image border. Only available in train mode. Defaults to 128.

• mean (sequence, optional) – Mean values of 3 channels.

• std (sequence, optional) – Std values of 3 channels.

• to_rgb (bool, optional) – Whether to convert the image from BGR to RGB.

• test_mode (bool) – whether involve random variables in transform. In train mode, crop_size is fixed, center coords and ratio is random selected from predefined lists. In test mode, crop_size is image’s original shape, center coords and ratio is fixed. Defaults to False.

• test_pad_mode (tuple, optional) –

  padding method and padding shape value, only available in test mode. Default is using ‘logical_or’ with 127 as padding shape value.

  • ’logical_or’: final_shape = input_shape | padding_shape_value

  • ’size_divisor’: final_shape = int( ceil(input_shape / padding_shape_value) * padding_shape_value)

  Defaults to (‘logical_or’, 127).

• test_pad_add_pix (int) – Extra padding pixel in test mode. Defaults to 0.

• bbox_clip_border (bool) – Whether clip the objects outside the border of the image. Defaults to True.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.RandomCrop(crop_size: tuple, crop_type: str = 'absolute', allow_negative_crop: bool = False, recompute_bbox: bool = False, bbox_clip_border: bool = True)

Random crop the image & bboxes & masks.

The absolute crop_size is sampled based on crop_type and image_size, then the cropped results are generated.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_ignore_flags (bool) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• img_shape

• gt_bboxes (optional)

• gt_bboxes_labels (optional)

• gt_masks (optional)

• gt_ignore_flags (optional)

• gt_seg_map (optional)

• gt_instances_ids (options, only used in MOT/VIS)

Added Keys:

• homography_matrix

Parameters

• crop_size (tuple) – The relative ratio or absolute pixels of (width, height).

• crop_type (str, optional) – One of “relative_range”, “relative”, “absolute”, “absolute_range”. “relative” randomly crops (h * crop_size[0], w * crop_size[1]) part from an input of size (h, w). “relative_range” uniformly samples relative crop size from range [crop_size[0], 1] and [crop_size[1], 1] for height and width respectively. “absolute” crops from an input with absolute size (crop_size[0], crop_size[1]). “absolute_range” uniformly samples crop_h in range [crop_size[0], min(h, crop_size[1])] and crop_w in range [crop_size[0], min(w, crop_size[1])]. Defaults to “absolute”.

• allow_negative_crop (bool, optional) – Whether to allow a crop that does not contain any bbox area. Defaults to False.

• recompute_bbox (bool, optional) – Whether to re-compute the boxes based on cropped instance masks. Defaults to False.

• bbox_clip_border (bool, optional) – Whether clip the objects outside the border of the image. Defaults to True.

Note

• If the image is smaller than the absolute crop size, return the

  original image.

• The keys for bboxes, labels and masks must be aligned. That is, gt_bboxes corresponds to gt_labels and gt_masks, and gt_bboxes_ignore corresponds to gt_labels_ignore and gt_masks_ignore.

• If the crop does not contain any gt-bbox region and allow_negative_crop is set to False, skip this image.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.RandomErasing(n_patches: Union[int, Tuple[int, int]], ratio: Union[float, Tuple[float, float]], squared: bool = True, bbox_erased_thr: float = 0.9, img_border_value: Union[int, float, tuple] = 128, mask_border_value: int = 0, seg_ignore_label: int = 255)

RandomErasing operation.

Random Erasing randomly selects a rectangle region in an image and erases its pixels with random values. RandomErasing.

Required Keys:

• img

• gt_bboxes (HorizontalBoxes[torch.float32]) (optional)

• gt_bboxes_labels (np.int64) (optional)

• gt_ignore_flags (bool) (optional)

• gt_masks (BitmapMasks) (optional)

Modified Keys: - img - gt_bboxes (optional) - gt_bboxes_labels (optional) - gt_ignore_flags (optional) - gt_masks (optional)

Parameters

• n_patches (int or tuple[int, int]) – Number of regions to be dropped. If it is given as a tuple, number of patches will be randomly selected from the closed interval [n_patches[0], n_patches[1]].

• ratio (float or tuple[float, float]) – The ratio of erased regions. It can be float to use a fixed ratio or tuple[float, float] to randomly choose ratio from the interval.

• squared (bool) – Whether to erase square region. Defaults to True.

• bbox_erased_thr (float) – The threshold for the maximum area proportion of the bbox to be erased. When the proportion of the area where the bbox is erased is greater than the threshold, the bbox will be removed. Defaults to 0.9.

• img_border_value (int or float or tuple) – The filled values for image border. If float, the same fill value will be used for all the three channels of image. If tuple, it should be 3 elements. Defaults to 128.

• mask_border_value (int) – The fill value used for masks. Defaults to 0.

• seg_ignore_label (int) – The fill value used for segmentation map. Note this value must equals ignore_label in semantic_head of the corresponding config. Defaults to 255.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.RandomFlip(prob: Optional[Union[float, Iterable[float]]] = None, direction: Union[str, Sequence[Optional[str]]] = 'horizontal', swap_seg_labels: Optional[Sequence] = None)

Flip the image & bbox & mask & segmentation map. Added or Updated keys: flip, flip_direction, img, gt_bboxes, and gt_seg_map. There are 3 flip modes:

• prob is float, direction is string: the image will be

  direction``ly flipped with probability of ``prob . E.g., prob=0.5, direction='horizontal', then image will be horizontally flipped with probability of 0.5.

• prob is float, direction is list of string: the image will

  be direction[i]``ly flipped with probability of ``prob/len(direction). E.g., prob=0.5, direction=['horizontal', 'vertical'], then image will be horizontally flipped with probability of 0.25, vertically with probability of 0.25.

• prob is list of float, direction is list of string:

  given len(prob) == len(direction), the image will be direction[i]``ly flipped with probability of ``prob[i]. E.g., prob=[0.3, 0.5], direction=['horizontal', 'vertical'], then image will be horizontally flipped with probability of 0.3, vertically with probability of 0.5.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• gt_bboxes

• gt_masks

• gt_seg_map

Added Keys:

• flip

• flip_direction

• homography_matrix

Parameters

• prob (float | list[float], optional) – The flipping probability. Defaults to None.

• direction (str | list[str]) – The flipping direction. Options If input is a list, the length must equal prob. Each element in prob indicates the flip probability of corresponding direction. Defaults to ‘horizontal’.

class mmdet.datasets.transforms.RandomFlip_GLIP(prob: Optional[Union[float, Iterable[float]]] = None, direction: Union[str, Sequence[Optional[str]]] = 'horizontal', swap_seg_labels: Optional[Sequence] = None)

Flip the image & bboxes & masks & segs horizontally or vertically.

When using horizontal flipping, the corresponding bbox x-coordinate needs to be additionally subtracted by one.

class mmdet.datasets.transforms.RandomOrder(transforms: Union[Dict, Callable[[Dict], Dict], Sequence[Union[Dict, Callable[[Dict], Dict]]]])

Shuffle the transform Sequence.

transform(results: Dict) → Optional[Dict]

Transform function to apply transforms in random order.

Parameters

results (dict) – A result dict contains the results to transform.

Returns

Transformed results.

Return type

dict or None

class mmdet.datasets.transforms.RandomSamplingNegPos(tokenizer_name, num_sample_negative=85, max_tokens=256, full_sampling_prob=0.5, label_map_file=None)

transform(results: dict) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.RandomShift(prob: float = 0.5, max_shift_px: int = 32, filter_thr_px: int = 1)

Shift the image and box given shift pixels and probability.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32])

• gt_bboxes_labels (np.int64)

• gt_ignore_flags (bool) (optional)

Modified Keys:

• img

• gt_bboxes

• gt_bboxes_labels

• gt_ignore_flags (bool) (optional)

Parameters

• prob (float) – Probability of shifts. Defaults to 0.5.

• max_shift_px (int) – The max pixels for shifting. Defaults to 32.

• filter_thr_px (int) – The width and height threshold for filtering. The bbox and the rest of the targets below the width and height threshold will be filtered. Defaults to 1.

transform(results: dict, *args, **kwargs) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.Resize(scale: Optional[Union[int, Tuple[int, int]]] = None, scale_factor: Optional[Union[float, Tuple[float, float]]] = None, keep_ratio: bool = False, clip_object_border: bool = True, backend: str = 'cv2', interpolation='bilinear')

Resize images & bbox & seg.

This transform resizes the input image according to scale or scale_factor. Bboxes, masks, and seg map are then resized with the same scale factor. if scale and scale_factor are both set, it will use scale to resize.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• img_shape

• gt_bboxes

• gt_masks

• gt_seg_map

Added Keys:

• scale

• scale_factor

• keep_ratio

• homography_matrix

Parameters

• scale (int or tuple) – Images scales for resizing. Defaults to None

• scale_factor (float or tuple[float]) – Scale factors for resizing. Defaults to None.

• keep_ratio (bool) – Whether to keep the aspect ratio when resizing the image. Defaults to False.

• clip_object_border (bool) – Whether to clip the objects outside the border of the image. In some dataset like MOT17, the gt bboxes are allowed to cross the border of images. Therefore, we don’t need to clip the gt bboxes in these cases. Defaults to True.

• backend (str) – Image resize backend, choices are ‘cv2’ and ‘pillow’. These two backends generates slightly different results. Defaults to ‘cv2’.

• interpolation (str) – Interpolation method, accepted values are “nearest”, “bilinear”, “bicubic”, “area”, “lanczos” for ‘cv2’ backend, “nearest”, “bilinear” for ‘pillow’ backend. Defaults to ‘bilinear’.

transform(results: dict, *args, **kwargs) → dict

Transform function to resize images, bounding boxes, semantic segmentation map and keypoints.

Parameters

results (dict) – Result dict from loading pipeline.

Returns

Resized results, ‘img’, ‘gt_bboxes’, ‘gt_seg_map’, ‘gt_keypoints’, ‘scale’, ‘scale_factor’, ‘img_shape’, and ‘keep_ratio’ keys are updated in result dict.

Return type

dict

class mmdet.datasets.transforms.ResizeShortestEdge(scale: Union[int, Tuple[int, int]], max_size: Optional[int] = None, resize_type: str = 'Resize', **resize_kwargs)

Resize the image and mask while keeping the aspect ratio unchanged.

Modified from https://github.com/facebookresearch/detectron2/blob/main/detectron2/data/transforms/augmentation_impl.py#L130 # noqa:E501

This transform attempts to scale the shorter edge to the given scale, as long as the longer edge does not exceed max_size. If max_size is reached, then downscale so that the longer edge does not exceed max_size.

Required Keys:

• img

• gt_seg_map (optional)

Modified Keys:

• img

• img_shape

• gt_seg_map (optional))

Added Keys:

• scale

• scale_factor

• keep_ratio

Parameters

• scale (Union[int, Tuple[int, int]]) – The target short edge length. If it’s tuple, will select the min value as the short edge length.

• max_size (int) – The maximum allowed longest edge length.

transform(results: dict) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

class mmdet.datasets.transforms.Rotate(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.0, max_mag: float = 30.0, reversal_prob: float = 0.5, img_border_value: Union[int, float, tuple] = 128, mask_border_value: int = 0, seg_ignore_label: int = 255, interpolation: str = 'bilinear')

Rotate the images, bboxes, masks and segmentation map.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• gt_bboxes

• gt_masks

• gt_seg_map

Added Keys:

• homography_matrix

Parameters

• prob (float) – The probability for perform transformation and should be in range 0 to 1. Defaults to 1.0.

• level (int, optional) – The level should be in range [0, _MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The maximum angle for rotation. Defaults to 0.0.

• max_mag (float) – The maximum angle for rotation. Defaults to 30.0.

• reversal_prob (float) – The probability that reverses the rotation magnitude. Should be in range [0,1]. Defaults to 0.5.

• img_border_value (int | float | tuple) – The filled values for image border. If float, the same fill value will be used for all the three channels of image. If tuple, it should be 3 elements. Defaults to 128.

• mask_border_value (int) – The fill value used for masks. Defaults to 0.

• seg_ignore_label (int) – The fill value used for segmentation map. Note this value must equals ignore_label in semantic_head of the corresponding config. Defaults to 255.

• interpolation (str) – Interpolation method, accepted values are “nearest”, “bilinear”, “bicubic”, “area”, “lanczos” for ‘cv2’ backend, “nearest”, “bilinear” for ‘pillow’ backend. Defaults to ‘bilinear’.

class mmdet.datasets.transforms.SegRescale(scale_factor: float = 1, backend: str = 'cv2')

Rescale semantic segmentation maps.

This transform rescale the gt_seg_map according to scale_factor.

Required Keys:

• gt_seg_map

Modified Keys:

• gt_seg_map

Parameters

• scale_factor (float) – The scale factor of the final output. Defaults to 1.

• backend (str) – Image rescale backend, choices are ‘cv2’ and ‘pillow’. These two backends generates slightly different results. Defaults to ‘cv2’.

transform(results: dict) → dict

Transform function to scale the semantic segmentation map.

Parameters

results (dict) – Result dict from loading pipeline.

Returns

Result dict with semantic segmentation map scaled.

Return type

dict

class mmdet.datasets.transforms.Sharpness(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.1, max_mag: float = 1.9)

Adjust images sharpness. A positive magnitude would enhance the sharpness and a negative magnitude would make the image blurry. A magnitude=0 gives the origin img.

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing Sharpness transformation. Defaults to 1.0.

• level (int, optional) – Should be in range [0,_MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum magnitude for Sharpness transformation. Defaults to 0.1.

• max_mag (float) – The maximum magnitude for Sharpness transformation. Defaults to 1.9.

class mmdet.datasets.transforms.ShearX(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.0, max_mag: float = 30.0, reversal_prob: float = 0.5, img_border_value: Union[int, float, tuple] = 128, mask_border_value: int = 0, seg_ignore_label: int = 255, interpolation: str = 'bilinear')

Shear the images, bboxes, masks and segmentation map horizontally.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• gt_bboxes

• gt_masks

• gt_seg_map

Added Keys:

• homography_matrix

Parameters

• prob (float) – The probability for performing Shear and should be in range [0, 1]. Defaults to 1.0.

• level (int, optional) – The level should be in range [0, _MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum angle for the horizontal shear. Defaults to 0.0.

• max_mag (float) – The maximum angle for the horizontal shear. Defaults to 30.0.

• reversal_prob (float) – The probability that reverses the horizontal shear magnitude. Should be in range [0,1]. Defaults to 0.5.

• img_border_value (int | float | tuple) – The filled values for image border. If float, the same fill value will be used for all the three channels of image. If tuple, it should be 3 elements. Defaults to 128.

• mask_border_value (int) – The fill value used for masks. Defaults to 0.

• seg_ignore_label (int) – The fill value used for segmentation map. Note this value must equals ignore_label in semantic_head of the corresponding config. Defaults to 255.

• interpolation (str) – Interpolation method, accepted values are “nearest”, “bilinear”, “bicubic”, “area”, “lanczos” for ‘cv2’ backend, “nearest”, “bilinear” for ‘pillow’ backend. Defaults to ‘bilinear’.

class mmdet.datasets.transforms.ShearY(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.0, max_mag: float = 30.0, reversal_prob: float = 0.5, img_border_value: Union[int, float, tuple] = 128, mask_border_value: int = 0, seg_ignore_label: int = 255, interpolation: str = 'bilinear')

Shear the images, bboxes, masks and segmentation map vertically.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• gt_bboxes

• gt_masks

• gt_seg_map

Added Keys:

• homography_matrix

Parameters

• prob (float) – The probability for performing ShearY and should be in range [0, 1]. Defaults to 1.0.

• level (int, optional) – The level should be in range [0,_MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum angle for the vertical shear. Defaults to 0.0.

• max_mag (float) – The maximum angle for the vertical shear. Defaults to 30.0.

• reversal_prob (float) – The probability that reverses the vertical shear magnitude. Should be in range [0,1]. Defaults to 0.5.

• img_border_value (int | float | tuple) – The filled values for image border. If float, the same fill value will be used for all the three channels of image. If tuple, it should be 3 elements. Defaults to 128.

• mask_border_value (int) – The fill value used for masks. Defaults to 0.

• seg_ignore_label (int) – The fill value used for segmentation map. Note this value must equals ignore_label in semantic_head of the corresponding config. Defaults to 255.

• interpolation (str) – Interpolation method, accepted values are “nearest”, “bilinear”, “bicubic”, “area”, “lanczos” for ‘cv2’ backend, “nearest”, “bilinear” for ‘pillow’ backend. Defaults to ‘bilinear’.

class mmdet.datasets.transforms.Solarize(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.0, max_mag: float = 256.0)

Solarize images (Invert all pixels above a threshold value of magnitude.).

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing Solarize transformation. Defaults to 1.0.

• level (int, optional) – Should be in range [0,_MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum magnitude for Solarize transformation. Defaults to 0.0.

• max_mag (float) – The maximum magnitude for Solarize transformation. Defaults to 256.0.

class mmdet.datasets.transforms.SolarizeAdd(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.0, max_mag: float = 110.0)

SolarizeAdd images. For each pixel in the image that is less than 128, add an additional amount to it decided by the magnitude.

Required Keys:

• img

Modified Keys:

• img

Parameters

• prob (float) – The probability for performing SolarizeAdd transformation. Defaults to 1.0.

• level (int, optional) – Should be in range [0,_MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum magnitude for SolarizeAdd transformation. Defaults to 0.0.

• max_mag (float) – The maximum magnitude for SolarizeAdd transformation. Defaults to 110.0.

class mmdet.datasets.transforms.ToTensor(keys)

Convert some results to torch.Tensor by given keys.

Parameters

keys (Sequence[str]) – Keys that need to be converted to Tensor.

class mmdet.datasets.transforms.TranslateX(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.0, max_mag: float = 0.1, reversal_prob: float = 0.5, img_border_value: Union[int, float, tuple] = 128, mask_border_value: int = 0, seg_ignore_label: int = 255, interpolation: str = 'bilinear')

Translate the images, bboxes, masks and segmentation map horizontally.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• gt_bboxes

• gt_masks

• gt_seg_map

Added Keys:

• homography_matrix

Parameters

• prob (float) – The probability for perform transformation and should be in range 0 to 1. Defaults to 1.0.

• level (int, optional) – The level should be in range [0, _MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum pixel’s offset ratio for horizontal translation. Defaults to 0.0.

• max_mag (float) – The maximum pixel’s offset ratio for horizontal translation. Defaults to 0.1.

• reversal_prob (float) – The probability that reverses the horizontal translation magnitude. Should be in range [0,1]. Defaults to 0.5.

• img_border_value (int | float | tuple) – The filled values for image border. If float, the same fill value will be used for all the three channels of image. If tuple, it should be 3 elements. Defaults to 128.

• mask_border_value (int) – The fill value used for masks. Defaults to 0.

• seg_ignore_label (int) – The fill value used for segmentation map. Note this value must equals ignore_label in semantic_head of the corresponding config. Defaults to 255.

• interpolation (str) – Interpolation method, accepted values are “nearest”, “bilinear”, “bicubic”, “area”, “lanczos” for ‘cv2’ backend, “nearest”, “bilinear” for ‘pillow’ backend. Defaults to ‘bilinear’.

class mmdet.datasets.transforms.TranslateY(prob: float = 1.0, level: Optional[int] = None, min_mag: float = 0.0, max_mag: float = 0.1, reversal_prob: float = 0.5, img_border_value: Union[int, float, tuple] = 128, mask_border_value: int = 0, seg_ignore_label: int = 255, interpolation: str = 'bilinear')

Translate the images, bboxes, masks and segmentation map vertically.

Required Keys:

• img

• gt_bboxes (BaseBoxes[torch.float32]) (optional)

• gt_masks (BitmapMasks | PolygonMasks) (optional)

• gt_seg_map (np.uint8) (optional)

Modified Keys:

• img

• gt_bboxes

• gt_masks

• gt_seg_map

Added Keys:

• homography_matrix

Parameters

• prob (float) – The probability for perform transformation and should be in range 0 to 1. Defaults to 1.0.

• level (int, optional) – The level should be in range [0, _MAX_LEVEL]. If level is None, it will generate from [0, _MAX_LEVEL] randomly. Defaults to None.

• min_mag (float) – The minimum pixel’s offset ratio for vertical translation. Defaults to 0.0.

• max_mag (float) – The maximum pixel’s offset ratio for vertical translation. Defaults to 0.1.

• reversal_prob (float) – The probability that reverses the vertical translation magnitude. Should be in range [0,1]. Defaults to 0.5.

• img_border_value (int | float | tuple) – The filled values for image border. If float, the same fill value will be used for all the three channels of image. If tuple, it should be 3 elements. Defaults to 128.

• mask_border_value (int) – The fill value used for masks. Defaults to 0.

• seg_ignore_label (int) – The fill value used for segmentation map. Note this value must equals ignore_label in semantic_head of the corresponding config. Defaults to 255.

• interpolation (str) – Interpolation method, accepted values are “nearest”, “bilinear”, “bicubic”, “area”, “lanczos” for ‘cv2’ backend, “nearest”, “bilinear” for ‘pillow’ backend. Defaults to ‘bilinear’.

class mmdet.datasets.transforms.Transpose(keys, order)

Transpose some results by given keys.

Parameters

• keys (Sequence[str]) – Keys of results to be transposed.

• order (Sequence[int]) – Order of transpose.

class mmdet.datasets.transforms.UniformRefFrameSample(num_ref_imgs: int = 1, frame_range: Union[int, List[int]] = 10, filter_key_img: bool = True, collect_video_keys: List[str] = ['video_id', 'video_length'])

Uniformly sample reference frames.

Parameters

• num_ref_imgs (int) – Number of reference frames to be sampled.

• frame_range (int | list[int]) – Range of frames to be sampled around key frame. If int, the range is [-frame_range, frame_range]. Defaults to 10.

• filter_key_img (bool) – Whether to filter the key frame when sampling reference frames. Defaults to True.

• collect_video_keys (list[str]) – The keys of video info to be collected.

sampling_frames(video_length: int, key_frame_id: int)

Sampling frames.

Parameters

• video_length (int) – The length of the video.

• key_frame_id (int) – The key frame id.

Returns

The sampled frame indices.

Return type

list[int]

transform(video_infos: dict) → Optional[Dict[str, List]]

Transform the video information.

Parameters

video_infos (dict) – The whole video information.

Returns

The data information of the sampled frames.

Return type

dict

class mmdet.datasets.transforms.YOLOXHSVRandomAug(hue_delta: int = 5, saturation_delta: int = 30, value_delta: int = 30)

Apply HSV augmentation to image sequentially. It is referenced from https://github.com/Megvii- BaseDetection/YOLOX/blob/main/yolox/data/data_augment.py#L21.

Required Keys:

• img

Modified Keys:

• img

Parameters

• hue_delta (int) – delta of hue. Defaults to 5.

• saturation_delta (int) – delta of saturation. Defaults to 30.

• value_delta (int) – delat of value. Defaults to 30.

transform(results: dict) → dict

The transform function. All subclass of BaseTransform should override this method.

This function takes the result dict as the input, and can add new items to the dict or modify existing items in the dict. And the result dict will be returned in the end, which allows to concate multiple transforms into a pipeline.

Parameters

results (dict) – The result dict.

Returns

The result dict.

Return type

dict

mmdet.engine

hooks

class mmdet.engine.hooks.CheckInvalidLossHook(interval: int = 50)

Check invalid loss hook.

This hook will regularly check whether the loss is valid during training.

Parameters

interval (int) – Checking interval (every k iterations). Default: 50.

after_train_iter(runner: Runner, batch_idx: int, data_batch: Optional[dict] = None, outputs: Optional[dict] = None) → None

Regularly check whether the loss is valid every n iterations.

Parameters

• runner (Runner) – The runner of the training process.

• batch_idx (int) – The index of the current batch in the train loop.

• data_batch (dict, Optional) – Data from dataloader. Defaults to None.

• outputs (dict, Optional) – Outputs from model. Defaults to None.

class mmdet.engine.hooks.DetVisualizationHook(draw: bool = False, interval: int = 50, score_thr: float = 0.3, show: bool = False, wait_time: float = 0.0, test_out_dir: Optional[str] = None, backend_args: Optional[dict] = None)

Detection Visualization Hook. Used to visualize validation and testing process prediction results.

In the testing phase:

1. If show is True, it means that only the prediction results are

   visualized without storing data, so vis_backends needs to be excluded.

2. If test_out_dir is specified, it means that the prediction results

   need to be saved to test_out_dir. In order to avoid vis_backends also storing data, so vis_backends needs to be excluded.

3. vis_backends takes effect if the user does not specify show

   and test_out_dir`. You can set vis_backends to WandbVisBackend or TensorboardVisBackend to store the prediction result in Wandb or Tensorboard.

Parameters

• draw (bool) – whether to draw prediction results. If it is False, it means that no drawing will be done. Defaults to False.

• interval (int) – The interval of visualization. Defaults to 50.

• score_thr (float) – The threshold to visualize the bboxes and masks. Defaults to 0.3.

• show (bool) – Whether to display the drawn image. Default to False.

• wait_time (float) – The interval of show (s). Defaults to 0.

• test_out_dir (str, optional) – directory where painted images will be saved in testing process.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

after_test_iter(runner: Runner, batch_idx: int, data_batch: dict, outputs: Sequence[DetDataSample]) → None

Run after every testing iterations.

Parameters

• runner (Runner) – The runner of the testing process.

• batch_idx (int) – The index of the current batch in the val loop.

• data_batch (dict) – Data from dataloader.

• outputs (Sequence[DetDataSample]) – A batch of data samples that contain annotations and predictions.

after_val_iter(runner: Runner, batch_idx: int, data_batch: dict, outputs: Sequence[DetDataSample]) → None

Run after every self.interval validation iterations.

Parameters

• runner (Runner) – The runner of the validation process.

• batch_idx (int) – The index of the current batch in the val loop.

• data_batch (dict) – Data from dataloader.

• outputs (Sequence[DetDataSample]]) – A batch of data samples that contain annotations and predictions.

class mmdet.engine.hooks.GroundingVisualizationHook(draw: bool = False, interval: int = 50, score_thr: float = 0.3, show: bool = False, wait_time: float = 0.0, test_out_dir: Optional[str] = None, backend_args: Optional[dict] = None)

after_test_iter(runner: Runner, batch_idx: int, data_batch: dict, outputs: Sequence[DetDataSample]) → None

Run after every testing iterations.

Parameters

• runner (Runner) – The runner of the testing process.

• batch_idx (int) – The index of the current batch in the val loop.

• data_batch (dict) – Data from dataloader.

• outputs (Sequence[DetDataSample]) – A batch of data samples that contain annotations and predictions.

class mmdet.engine.hooks.MeanTeacherHook(momentum: float = 0.001, interval: int = 1, skip_buffer=True)

Mean Teacher Hook.

Mean Teacher is an efficient semi-supervised learning method in Mean Teacher. This method requires two models with exactly the same structure, as the student model and the teacher model, respectively. The student model updates the parameters through gradient descent, and the teacher model updates the parameters through exponential moving average of the student model. Compared with the student model, the teacher model is smoother and accumulates more knowledge.

Parameters

• momentum (float) –

  The momentum used for updating teacher’s parameter.

  Teacher’s parameter are updated with the formula:

  teacher = (1-momentum) * teacher + momentum * student.

  Defaults to 0.001.

• interval (int) – Update teacher’s parameter every interval iteration. Defaults to 1.

• skip_buffers (bool) – Whether to skip the model buffers, such as batchnorm running stats (running_mean, running_var), it does not perform the ema operation. Default to True.

after_train_iter(runner: Runner, batch_idx: int, data_batch: Optional[dict] = None, outputs: Optional[dict] = None) → None

Update teacher’s parameter every self.interval iterations.

before_train(runner: Runner) → None

To check that teacher model and student model exist.

momentum_update(model: Module, momentum: float) → None

Compute the moving average of the parameters using exponential moving average.

class mmdet.engine.hooks.MemoryProfilerHook(interval: int = 50)

Memory profiler hook recording memory information including virtual memory, swap memory, and the memory of the current process.

Parameters

interval (int) – Checking interval (every k iterations). Default: 50.

after_test_iter(runner: Runner, batch_idx: int, data_batch: Optional[dict] = None, outputs: Optional[Sequence[DetDataSample]] = None) → None

Regularly record memory information.

Parameters

• runner (Runner) – The runner of the testing process.

• batch_idx (int) – The index of the current batch in the test loop.

• data_batch (dict, optional) – Data from dataloader. Defaults to None.

• outputs (Sequence[DetDataSample], optional) – Outputs from model. Defaults to None.

after_train_iter(runner: Runner, batch_idx: int, data_batch: Optional[dict] = None, outputs: Optional[dict] = None) → None

Regularly record memory information.

Parameters

• runner (Runner) – The runner of the training process.

• batch_idx (int) – The index of the current batch in the train loop.

• data_batch (dict, optional) – Data from dataloader. Defaults to None.

• outputs (dict, optional) – Outputs from model. Defaults to None.

after_val_iter(runner: Runner, batch_idx: int, data_batch: Optional[dict] = None, outputs: Optional[Sequence[DetDataSample]] = None) → None

Regularly record memory information.

Parameters

• runner (Runner) – The runner of the validation process.

• batch_idx (int) – The index of the current batch in the val loop.

• data_batch (dict, optional) – Data from dataloader. Defaults to None.

• outputs (Sequence[DetDataSample], optional) – Outputs from model. Defaults to None.

class mmdet.engine.hooks.NumClassCheckHook

Check whether the num_classes in head matches the length of classes in dataset.metainfo.

before_train_epoch(runner: Runner) → None

Check whether the training dataset is compatible with head.

Parameters

runner (Runner) – The runner of the training or evaluation process.

before_val_epoch(runner: Runner) → None

Check whether the dataset in val epoch is compatible with head.

Parameters

runner (Runner) – The runner of the training or evaluation process.

class mmdet.engine.hooks.PipelineSwitchHook(switch_epoch, switch_pipeline)

Switch data pipeline at switch_epoch.

Parameters

• switch_epoch (int) – switch pipeline at this epoch.

• switch_pipeline (list[dict]) – the pipeline to switch to.

before_train_epoch(runner)

Switch pipeline.

class mmdet.engine.hooks.SetEpochInfoHook

Set runner’s epoch information to the model.

before_train_epoch(runner)

All subclasses should override this method, if they need any operations before each training epoch.

Parameters

runner (Runner) – The runner of the training process.

class mmdet.engine.hooks.SyncNormHook

Synchronize Norm states before validation, currently used in YOLOX.

before_val_epoch(runner)

Synchronizing norm.

class mmdet.engine.hooks.TrackVisualizationHook(draw: bool = False, frame_interval: int = 30, score_thr: float = 0.3, show: bool = False, wait_time: float = 0.0, test_out_dir: Optional[str] = None, backend_args: Optional[dict] = None)

Tracking Visualization Hook. Used to visualize validation and testing process prediction results.

In the testing phase:

1. If show is True, it means that only the prediction results are

   visualized without storing data, so vis_backends needs to be excluded.

2. If test_out_dir is specified, it means that the prediction results

   need to be saved to test_out_dir. In order to avoid vis_backends also storing data, so vis_backends needs to be excluded.

3. vis_backends takes effect if the user does not specify show

   and test_out_dir`. You can set vis_backends to WandbVisBackend or TensorboardVisBackend to store the prediction result in Wandb or Tensorboard.

Parameters

• draw (bool) – whether to draw prediction results. If it is False, it means that no drawing will be done. Defaults to False.

• frame_interval (int) – The interval of visualization. Defaults to 30.

• score_thr (float) – The threshold to visualize the bboxes and masks. Defaults to 0.3.

• show (bool) – Whether to display the drawn image. Default to False.

• wait_time (float) – The interval of show (s). Defaults to 0.

• test_out_dir (str, optional) – directory where painted images will be saved in testing process.

• backend_args (dict) – Arguments to instantiate a file client. Defaults to None.

after_test_iter(runner: Runner, batch_idx: int, data_batch: dict, outputs: Sequence[TrackDataSample]) → None

Run after every testing iteration.

Parameters

• runner (Runner) – The runner of the testing process.

• batch_idx (int) – The index of the current batch in the test loop.

• data_batch (dict) – Data from dataloader.

• outputs (Sequence[TrackDataSample]) – Outputs from model.

after_val_iter(runner: Runner, batch_idx: int, data_batch: dict, outputs: Sequence[TrackDataSample]) → None

Run after every self.interval validation iteration.

Parameters

• runner (Runner) – The runner of the validation process.

• batch_idx (int) – The index of the current batch in the val loop.

• data_batch (dict) – Data from dataloader.

• outputs (Sequence[TrackDataSample]) – Outputs from model.

visualize_single_image(img_data_sample: DetDataSample, step: int) → None

Parameters

• img_data_sample (DetDataSample) – single image output.

• step (int) – The index of the current image.

class mmdet.engine.hooks.TrainAugmentDetVisualizationHook(draw: bool = False, interval: int = 50, show: bool = False, wait_time: float = 0.0, backend_args: Optional[dict] = None)

Detection Visualization Hook. Used to visualize train augmentation.

In the training phase:

1. If show is True, it means that only the prediction results are

   visualized without storing data, so vis_backends needs to be excluded.

2. vis_backends takes effect if the user does not specify show.

   You can set vis_backends to WandbVisBackend or TensorboardVisBackend to store the prediction result in Wandb or Tensorboard.

Parameters

• draw (bool) – whether to draw prediction results. If it is False, it means that no drawing will be done. Defaults to False.

• interval (int) – The interval of visualization. Defaults to 50.

• show (bool) – Whether to display the drawn image. Default to False.

• wait_time (float) – The interval of show (s). Defaults to 0.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

after_train_iter(runner: Runner, batch_idx: int, data_batch: Optional[dict] = None, outputs: Optional[dict] = None) → None

Regularly check whether the loss is valid every n iterations.

Parameters

• runner (Runner) – The runner of the training process.

• batch_idx (int) – The index of the current batch in the train loop.

• data_batch (dict, Optional) – Data from dataloader. Defaults to None.

• outputs (dict, Optional) – Outputs from model. Defaults to None.

class mmdet.engine.hooks.YOLOXModeSwitchHook(num_last_epochs: int = 15, skip_type_keys: Sequence[str] = ('Mosaic', 'RandomAffine', 'MixUp'))

Switch the mode of YOLOX during training.

This hook turns off the mosaic and mixup data augmentation and switches to use L1 loss in bbox_head.

Parameters

num_last_epochs – The number of latter epochs in the end of the training to close the data augmentation and switch to L1 loss. Defaults to 15.

before_train_epoch(runner) → None

Close mosaic and mixup augmentation and switches to use L1 loss.

optimizers

class mmdet.engine.optimizers.LearningRateDecayOptimizerConstructor(optim_wrapper_cfg: dict, paramwise_cfg: Optional[dict] = None)

add_params(params: List[dict], module: Module, **kwargs) → None

Add all parameters of module to the params list.

The parameters of the given module will be added to the list of param groups, with specific rules defined by paramwise_cfg.

Parameters

• params (list[dict]) – A list of param groups, it will be modified in place.

• module (nn.Module) – The module to be added.

runner

class mmdet.engine.runner.TeacherStudentValLoop(runner, dataloader: Union[DataLoader, Dict], evaluator: Union[Evaluator, Dict, List], fp16: bool = False)

Loop for validation of model teacher and student.

run()

Launch validation for model teacher and student.

schedulers

class mmdet.engine.schedulers.QuadraticWarmupLR(optimizer, *args, **kwargs)

Warm up the learning rate of each parameter group by quadratic formula.

Parameters

• optimizer (Optimizer) – Wrapped optimizer.

• begin (int) – Step at which to start updating the parameters. Defaults to 0.

• end (int) – Step at which to stop updating the parameters. Defaults to INF.

• last_step (int) – The index of last step. Used for resume without state dict. Defaults to -1.

• by_epoch (bool) – Whether the scheduled parameters are updated by epochs. Defaults to True.

• verbose (bool) – Whether to print the value for each update. Defaults to False.

class mmdet.engine.schedulers.QuadraticWarmupMomentum(optimizer, *args, **kwargs)

Warm up the momentum value of each parameter group by quadratic formula.

Parameters

• optimizer (Optimizer) – Wrapped optimizer.

• begin (int) – Step at which to start updating the parameters. Defaults to 0.

• end (int) – Step at which to stop updating the parameters. Defaults to INF.

• last_step (int) – The index of last step. Used for resume without state dict. Defaults to -1.

• by_epoch (bool) – Whether the scheduled parameters are updated by epochs. Defaults to True.

• verbose (bool) – Whether to print the value for each update. Defaults to False.

class mmdet.engine.schedulers.QuadraticWarmupParamScheduler(optimizer: Optimizer, param_name: str, begin: int = 0, end: int = 1000000000, last_step: int = -1, by_epoch: bool = True, verbose: bool = False)

Warm up the parameter value of each parameter group by quadratic formula:

\[X_{t} = X_{t-1} + \frac{2t+1}{{(end-begin)}^{2}} \times X_{base}\]

Parameters

• optimizer (Optimizer) – Wrapped optimizer.

• param_name (str) – Name of the parameter to be adjusted, such as lr, momentum.

• begin (int) – Step at which to start updating the parameters. Defaults to 0.

• end (int) – Step at which to stop updating the parameters. Defaults to INF.

• last_step (int) – The index of last step. Used for resume without state dict. Defaults to -1.

• by_epoch (bool) – Whether the scheduled parameters are updated by epochs. Defaults to True.

• verbose (bool) – Whether to print the value for each update. Defaults to False.

classmethod build_iter_from_epoch(*args, begin=0, end=1000000000, by_epoch=True, epoch_length=None, **kwargs)

Build an iter-based instance of this scheduler from an epoch-based config.

mmdet.evaluation

functional

mmdet.evaluation.functional.average_precision(recalls, precisions, mode='area')

Calculate average precision (for single or multiple scales).

Parameters

• recalls (ndarray) – shape (num_scales, num_dets) or (num_dets, )

• precisions (ndarray) – shape (num_scales, num_dets) or (num_dets, )

• mode (str) – ‘area’ or ‘11points’, ‘area’ means calculating the area under precision-recall curve, ‘11points’ means calculating the average precision of recalls at [0, 0.1, …, 1]

Returns

calculated average precision

Return type

float or ndarray

mmdet.evaluation.functional.bbox_overlaps(bboxes1, bboxes2, mode='iou', eps=1e-06, use_legacy_coordinate=False)

Calculate the ious between each bbox of bboxes1 and bboxes2.

Parameters

• bboxes1 (ndarray) – Shape (n, 4)

• bboxes2 (ndarray) – Shape (k, 4)

• mode (str) – IOU (intersection over union) or IOF (intersection over foreground)

• use_legacy_coordinate (bool) – Whether to use coordinate system in mmdet v1.x. which means width, height should be calculated as ‘x2 - x1 + 1` and ‘y2 - y1 + 1’ respectively. Note when function is used in VOCDataset, it should be True to align with the official implementation http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCdevkit_18-May-2011.tar Default: False.

Returns

Shape (n, k)

Return type

ious (ndarray)

mmdet.evaluation.functional.cityscapes_classes() → list

Class names of Cityscapes.

mmdet.evaluation.functional.coco_classes() → list

Class names of COCO.

mmdet.evaluation.functional.coco_panoptic_classes() → list

Class names of COCO panoptic.

mmdet.evaluation.functional.eval_map(det_results, annotations, scale_ranges=None, iou_thr=0.5, ioa_thr=None, dataset=None, logger=None, tpfp_fn=None, nproc=4, use_legacy_coordinate=False, use_group_of=False, eval_mode='area')

Evaluate mAP of a dataset.

Parameters

• det_results (list[list]) – [[cls1_det, cls2_det, …], …]. The outer list indicates images, and the inner list indicates per-class detected bboxes.

• annotations (list[dict]) –

  Ground truth annotations where each item of the list indicates an image. Keys of annotations are:

  • bboxes: numpy array of shape (n, 4)

  • labels: numpy array of shape (n, )

  • bboxes_ignore (optional): numpy array of shape (k, 4)

  • labels_ignore (optional): numpy array of shape (k, )

• scale_ranges (list[tuple] | None) – Range of scales to be evaluated, in the format [(min1, max1), (min2, max2), …]. A range of (32, 64) means the area range between (32**2, 64**2). Defaults to None.

• iou_thr (float) – IoU threshold to be considered as matched. Defaults to 0.5.

• ioa_thr (float | None) – IoA threshold to be considered as matched, which only used in OpenImages evaluation. Defaults to None.

• dataset (list[str] | str | None) – Dataset name or dataset classes, there are minor differences in metrics for different datasets, e.g. “voc”, “imagenet_det”, etc. Defaults to None.

• logger (logging.Logger | str | None) – The way to print the mAP summary. See mmengine.logging.print_log() for details. Defaults to None.

• tpfp_fn (callable | None) – The function used to determine true/ false positives. If None, tpfp_default() is used as default unless dataset is ‘det’ or ‘vid’ (tpfp_imagenet() in this case). If it is given as a function, then this function is used to evaluate tp & fp. Default None.

• nproc (int) – Processes used for computing TP and FP. Defaults to 4.

• use_legacy_coordinate (bool) – Whether to use coordinate system in mmdet v1.x. which means width, height should be calculated as ‘x2 - x1 + 1` and ‘y2 - y1 + 1’ respectively. Defaults to False.

• use_group_of (bool) – Whether to use group of when calculate TP and FP, which only used in OpenImages evaluation. Defaults to False.

• eval_mode (str) – ‘area’ or ‘11points’, ‘area’ means calculating the area under precision-recall curve, ‘11points’ means calculating the average precision of recalls at [0, 0.1, …, 1], PASCAL VOC2007 uses 11points as default evaluate mode, while others are ‘area’. Defaults to ‘area’.

Returns

(mAP, [dict, dict, …])

Return type

tuple

mmdet.evaluation.functional.eval_recalls(gts, proposals, proposal_nums=None, iou_thrs=0.5, logger=None, use_legacy_coordinate=False)

Calculate recalls.

Parameters

• gts (list[ndarray]) – a list of arrays of shape (n, 4)

• proposals (list[ndarray]) – a list of arrays of shape (k, 4) or (k, 5)

• proposal_nums (int | Sequence[int]) – Top N proposals to be evaluated.

• iou_thrs (float | Sequence[float]) – IoU thresholds. Default: 0.5.

• logger (logging.Logger | str | None) – The way to print the recall summary. See mmengine.logging.print_log() for details. Default: None.

• use_legacy_coordinate (bool) – Whether use coordinate system in mmdet v1.x. “1” was added to both height and width which means w, h should be computed as ‘x2 - x1 + 1` and ‘y2 - y1 + 1’. Default: False.

Returns

recalls of different ious and proposal nums

Return type

ndarray

mmdet.evaluation.functional.evaluateImgLists(prediction_list: list, groundtruth_list: list, args: CArgs, backend_args: Optional[dict] = None, dump_matches: bool = False) → dict

A wrapper of obj:``cityscapesscripts.evaluation.

evalInstanceLevelSemanticLabeling.evaluateImgLists``. Support loading groundtruth image from file backend. :param prediction_list: A list of prediction txt file. :type prediction_list: list :param groundtruth_list: A list of groundtruth image file. :type groundtruth_list: list :param args: A global object setting in

obj:cityscapesscripts.evaluation. evalInstanceLevelSemanticLabeling

Parameters

• backend_args (dict, optional) – Arguments to instantiate the preifx of uri corresponding backend. Defaults to None.

• dump_matches (bool) – whether dump matches.json. Defaults to False.

Returns

The computed metric.

Return type

dict

mmdet.evaluation.functional.get_classes(dataset) → list

Get class names of a dataset.

mmdet.evaluation.functional.imagenet_det_classes() → list

Class names of ImageNet Det.

mmdet.evaluation.functional.imagenet_vid_classes() → list

Class names of ImageNet VID.

mmdet.evaluation.functional.objects365v1_classes() → list

Class names of Objects365 V1.

mmdet.evaluation.functional.objects365v2_classes() → list

Class names of Objects365 V2.

mmdet.evaluation.functional.oid_challenge_classes() → list

Class names of Open Images Challenge.

mmdet.evaluation.functional.oid_v6_classes() → list

Class names of Open Images V6.

mmdet.evaluation.functional.plot_iou_recall(recalls, iou_thrs)

Plot IoU-Recalls curve.

Parameters

• recalls (ndarray or list) – shape (k,)

• iou_thrs (ndarray or list) – same shape as recalls

mmdet.evaluation.functional.plot_num_recall(recalls, proposal_nums)

Plot Proposal_num-Recalls curve.

Parameters

• recalls (ndarray or list) – shape (k,)

• proposal_nums (ndarray or list) – same shape as recalls

mmdet.evaluation.functional.pq_compute_multi_core(matched_annotations_list, gt_folder, pred_folder, categories, backend_args=None, nproc=32)

Evaluate the metrics of Panoptic Segmentation with multithreading.

Same as the function with the same name in panopticapi.

Parameters

• matched_annotations_list (list) – The matched annotation list. Each element is a tuple of annotations of the same image with the format (gt_anns, pred_anns).

• gt_folder (str) – The path of the ground truth images.

• pred_folder (str) – The path of the prediction images.

• categories (str) – The categories of the dataset.

• backend_args (object) – The file client of the dataset. If None, the backend will be set to local.

• nproc (int) – Number of processes for panoptic quality computing. Defaults to 32. When nproc exceeds the number of cpu cores, the number of cpu cores is used.

mmdet.evaluation.functional.pq_compute_single_core(proc_id, annotation_set, gt_folder, pred_folder, categories, backend_args=None, print_log=False)

The single core function to evaluate the metric of Panoptic Segmentation.

Same as the function with the same name in panopticapi. Only the function to load the images is changed to use the file client.

Parameters

• proc_id (int) – The id of the mini process.

• gt_folder (str) – The path of the ground truth images.

• pred_folder (str) – The path of the prediction images.

• categories (str) – The categories of the dataset.

• backend_args (object) – The Backend of the dataset. If None, the backend will be set to local.

• print_log (bool) – Whether to print the log. Defaults to False.

mmdet.evaluation.functional.print_map_summary(mean_ap, results, dataset=None, scale_ranges=None, logger=None)

Print mAP and results of each class.

A table will be printed to show the gts/dets/recall/AP of each class and the mAP.

Parameters

• mean_ap (float) – Calculated from eval_map().

• results (list[dict]) – Calculated from eval_map().

• dataset (list[str] | str | None) – Dataset name or dataset classes.

• scale_ranges (list[tuple] | None) – Range of scales to be evaluated.

• logger (logging.Logger | str | None) – The way to print the mAP summary. See mmengine.logging.print_log() for details. Defaults to None.

mmdet.evaluation.functional.print_recall_summary(recalls, proposal_nums, iou_thrs, row_idxs=None, col_idxs=None, logger=None)

Print recalls in a table.

Parameters

• recalls (ndarray) – calculated from bbox_recalls

• proposal_nums (ndarray or list) – top N proposals

• iou_thrs (ndarray or list) – iou thresholds

• row_idxs (ndarray) – which rows(proposal nums) to print

• col_idxs (ndarray) – which cols(iou thresholds) to print

• logger (logging.Logger | str | None) – The way to print the recall summary. See mmengine.logging.print_log() for details. Default: None.

mmdet.evaluation.functional.voc_classes() → list

Class names of PASCAL VOC.

metrics

class mmdet.evaluation.metrics.BaseVideoMetric(collect_device: str = 'cpu', prefix: Optional[str] = None, collect_dir: Optional[str] = None)

Base class for a metric in video task.

The metric first processes each batch of data_samples and predictions, and appends the processed results to the results list. Then it collects all results together from all ranks if distributed training is used. Finally, it computes the metrics of the entire dataset.

A subclass of class:BaseVideoMetric should assign a meaningful value to the class attribute default_prefix. See the argument prefix for details.

evaluate(size: int = 1) → dict

Evaluate the model performance of the whole dataset after processing all batches.

Parameters

size (int) – Length of the entire validation dataset.

Returns

Evaluation metrics dict on the val dataset. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions.

The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

class mmdet.evaluation.metrics.COCOCaptionMetric(ann_file: str, collect_device: str = 'cpu', prefix: Optional[str] = None)

Coco Caption evaluation wrapper.

Save the generated captions and transform into coco format. Calling COCO API for caption metrics.

Parameters

• ann_file (str) – the path for the COCO format caption ground truth json file, load for evaluations.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Should be modified according to the retrieval_type for unambiguous results. Defaults to TR.

compute_metrics(results: List)

Compute the metrics from processed results.

Parameters

results (dict) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

Dict

process(data_batch, data_samples)

Process one batch of data samples.

The processed results should be stored in self.results, which will be used to computed the metrics when all batches have been processed.

Parameters

• data_batch – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of outputs from the model.

class mmdet.evaluation.metrics.CULaneMetric(collect_device: str = 'cpu', prefix: Optional[str] = None, collect_dir: Optional[str] = None, iou_thresholds: Optional[Sequence[float]] = (0.5,))

compute_metrics(results) → dict

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: Any, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (Any) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of outputs from the model.

class mmdet.evaluation.metrics.CityScapesMetric(outfile_prefix: str, seg_prefix: Optional[str] = None, format_only: bool = False, collect_device: str = 'cpu', prefix: Optional[str] = None, dump_matches: bool = False, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None)

CityScapes metric for instance segmentation.

Parameters

• outfile_prefix (str) – The prefix of txt and png files. The txt and png file will be save in a directory whose path is “outfile_prefix.results/”.

• seg_prefix (str, optional) – Path to the directory which contains the cityscapes instance segmentation masks. It’s necessary when training and validation. It could be None when infer on test dataset. Defaults to None.

• format_only (bool) – Format the output results without perform evaluation. It is useful when you want to format the result to a specific format and submit it to the test server. Defaults to False.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

• dump_matches (bool) – Whether dump matches.json file during evaluating. Defaults to False.

• file_client_args (dict, optional) – Arguments to instantiate the corresponding backend in mmdet <= 3.0.0rc6. Defaults to None.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of

the metrics, and the values are corresponding results.

Return type

Dict[str, float]

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

class mmdet.evaluation.metrics.CocoMetric(ann_file: Optional[str] = None, metric: Union[str, List[str]] = 'bbox', classwise: bool = False, proposal_nums: Sequence[int] = (100, 300, 1000), iou_thrs: Optional[Union[float, Sequence[float]]] = None, metric_items: Optional[Sequence[str]] = None, format_only: bool = False, outfile_prefix: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, collect_device: str = 'cpu', prefix: Optional[str] = None, sort_categories: bool = False, use_mp_eval: bool = False)

COCO evaluation metric.

Evaluate AR, AP, and mAP for detection tasks including proposal/box detection and instance segmentation. Please refer to https://cocodataset.org/#detection-eval for more details.

Parameters

• ann_file (str, optional) – Path to the coco format annotation file. If not specified, ground truth annotations from the dataset will be converted to coco format. Defaults to None.

• metric (str | List[str]) – Metrics to be evaluated. Valid metrics include ‘bbox’, ‘segm’, ‘proposal’, and ‘proposal_fast’. Defaults to ‘bbox’.

• classwise (bool) – Whether to evaluate the metric class-wise. Defaults to False.

• proposal_nums (Sequence[int]) – Numbers of proposals to be evaluated. Defaults to (100, 300, 1000).

• iou_thrs (float | List[float], optional) – IoU threshold to compute AP and AR. If not specified, IoUs from 0.5 to 0.95 will be used. Defaults to None.

• metric_items (List[str], optional) – Metric result names to be recorded in the evaluation result. Defaults to None.

• format_only (bool) – Format the output results without perform evaluation. It is useful when you want to format the result to a specific format and submit it to the test server. Defaults to False.

• outfile_prefix (str, optional) – The prefix of json files. It includes the file path and the prefix of filename, e.g., “a/b/prefix”. If not specified, a temp file will be created. Defaults to None.

• file_client_args (dict, optional) – Arguments to instantiate the corresponding backend in mmdet <= 3.0.0rc6. Defaults to None.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

• sort_categories (bool) – Whether sort categories in annotations. Only used for Objects365V1Dataset. Defaults to False.

• use_mp_eval (bool) – Whether to use mul-processing evaluation

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

Dict[str, float]

fast_eval_recall(results: List[dict], proposal_nums: Sequence[int], iou_thrs: Sequence[float], logger: Optional[MMLogger] = None) → ndarray

Evaluate proposal recall with COCO’s fast_eval_recall.

Parameters

• results (List[dict]) – Results of the dataset.

• proposal_nums (Sequence[int]) – Proposal numbers used for evaluation.

• iou_thrs (Sequence[float]) – IoU thresholds used for evaluation.

• logger (MMLogger, optional) – Logger used for logging the recall summary.

Returns

Averaged recall results.

Return type

np.ndarray

gt_to_coco_json(gt_dicts: Sequence[dict], outfile_prefix: str) → str

Convert ground truth to coco format json file.

Parameters

• gt_dicts (Sequence[dict]) – Ground truth of the dataset.

• outfile_prefix (str) – The filename prefix of the json files. If the prefix is “somepath/xxx”, the json file will be named “somepath/xxx.gt.json”.

Returns

The filename of the json file.

Return type

str

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

results2json(results: Sequence[dict], outfile_prefix: str) → dict

Dump the detection results to a COCO style json file.

There are 3 types of results: proposals, bbox predictions, mask predictions, and they have different data types. This method will automatically recognize the type, and dump them to json files.

Parameters

• results (Sequence[dict]) – Testing results of the dataset.

• outfile_prefix (str) – The filename prefix of the json files. If the prefix is “somepath/xxx”, the json files will be named “somepath/xxx.bbox.json”, “somepath/xxx.segm.json”, “somepath/xxx.proposal.json”.

Returns

Possible keys are “bbox”, “segm”, “proposal”, and values are corresponding filenames.

Return type

dict

xyxy2xywh(bbox: ndarray) → list

Convert xyxy style bounding boxes to xywh style for COCO evaluation.

Parameters

bbox (numpy.ndarray) – The bounding boxes, shape (4, ), in xyxy order.

Returns

The converted bounding boxes, in xywh order.

Return type

list[float]

class mmdet.evaluation.metrics.CocoOccludedSeparatedMetric(*args, occluded_ann: str = 'https://www.robots.ox.ac.uk/~vgg/research/tpod/datasets/occluded_coco.pkl', separated_ann: str = 'https://www.robots.ox.ac.uk/~vgg/research/tpod/datasets/separated_coco.pkl', score_thr: float = 0.3, iou_thr: float = 0.75, metric: Union[str, List[str]] = ['bbox', 'segm'], **kwargs)

Metric of separated and occluded masks which presented in paper `A Tri- Layer Plugin to Improve Occluded Detection.

<https://arxiv.org/abs/2210.10046>`_.

Separated COCO and Occluded COCO are automatically generated subsets of COCO val dataset, collecting separated objects and partially occluded objects for a large variety of categories. In this way, we define occlusion into two major categories: separated and partially occluded.

• Separation: target object segmentation mask is separated into distinct regions by the occluder.

• Partial Occlusion: target object is partially occluded but the segmentation mask is connected.

These two new scalable real-image datasets are to benchmark a model’s capability to detect occluded objects of 80 common categories.

Please cite the paper if you use this dataset:

@article{zhan2022triocc,

title={A Tri-Layer Plugin to Improve Occluded Detection}, author={Zhan, Guanqi and Xie, Weidi and Zisserman, Andrew}, journal={British Machine Vision Conference}, year={2022}

}

Parameters

• occluded_ann (str) – Path to the occluded coco annotation file.

• separated_ann (str) – Path to the separated coco annotation file.

• score_thr (float) – Score threshold of the detection masks. Defaults to 0.3.

• iou_thr (float) – IoU threshold for the recall calculation. Defaults to 0.75.

• metric (str | List[str]) – Metrics to be evaluated. Valid metrics include ‘bbox’, ‘segm’, ‘proposal’, and ‘proposal_fast’. Defaults to ‘bbox’.

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

Dict[str, float]

compute_recall(result_dict: dict, gt_ann: list, is_occ: bool = True) → tuple

Compute the recall of occluded or separated masks.

Parameters

• result_dict (dict) – Processed mask results.

• gt_ann (list) – Occluded or separated coco annotations.

• is_occ (bool) – Whether the annotation is occluded mask. Defaults to True.

Returns

number of correct masks and the recall.

Return type

tuple

evaluate_occluded_separated(results: List[tuple]) → dict

Compute the recall of occluded and separated masks.

Parameters

results (list[tuple]) – Testing results of the dataset.

Returns

The recall of occluded and separated masks.

Return type

dict[str, float]

mask_iou(mask1: ndarray, mask2: ndarray) → ndarray

Compute IoU between two masks.

class mmdet.evaluation.metrics.CocoPanopticMetric(ann_file: Optional[str] = None, seg_prefix: Optional[str] = None, classwise: bool = False, format_only: bool = False, outfile_prefix: Optional[str] = None, nproc: int = 32, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, collect_device: str = 'cpu', prefix: Optional[str] = None)

COCO panoptic segmentation evaluation metric.

Evaluate PQ, SQ RQ for panoptic segmentation tasks. Please refer to https://cocodataset.org/#panoptic-eval for more details.

Parameters

• ann_file (str, optional) – Path to the coco format annotation file. If not specified, ground truth annotations from the dataset will be converted to coco format. Defaults to None.

• seg_prefix (str, optional) – Path to the directory which contains the coco panoptic segmentation mask. It should be specified when evaluate. Defaults to None.

• classwise (bool) – Whether to evaluate the metric class-wise. Defaults to False.

• outfile_prefix (str, optional) – The prefix of json files. It includes the file path and the prefix of filename, e.g., “a/b/prefix”. If not specified, a temp file will be created. It should be specified when format_only is True. Defaults to None.

• format_only (bool) – Format the output results without perform evaluation. It is useful when you want to format the result to a specific format and submit it to the test server. Defaults to False.

• nproc (int) – Number of processes for panoptic quality computing. Defaults to 32. When nproc exceeds the number of cpu cores, the number of cpu cores is used.

• file_client_args (dict, optional) – Arguments to instantiate the corresponding backend in mmdet <= 3.0.0rc6. Defaults to None.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) –

The processed results of each batch. There are two cases:

• When outfile_prefix is not provided, the elements in results are pq_stats which can be summed directly to get PQ.

• When outfile_prefix is provided, the elements in results are tuples like (gt, pred).

Returns

The computed metrics. The keys are the names of

the metrics, and the values are corresponding results.

Return type

Dict[str, float]

gt_to_coco_json(gt_dicts: Sequence[dict], outfile_prefix: str) → Tuple[str, str]

Convert ground truth to coco panoptic segmentation format json file.

Parameters

• gt_dicts (Sequence[dict]) – Ground truth of the dataset.

• outfile_prefix (str) – The filename prefix of the json file. If the prefix is “somepath/xxx”, the json file will be named “somepath/xxx.gt.json”.

Returns

The filename of the json file and the name of the directory which contains panoptic segmentation masks.

Return type

Tuple[str, str]

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

result2json(results: Sequence[dict], outfile_prefix: str) → Tuple[str, str]

Dump the panoptic results to a COCO style json file and a directory.

Parameters

• results (Sequence[dict]) – Testing results of the dataset.

• outfile_prefix (str) – The filename prefix of the json files and the directory.

Returns

The json file and the directory which contains panoptic segmentation masks. The filename of the json is

”somepath/xxx.panoptic.json” and name of the directory is “somepath/xxx.panoptic”.

Return type

Tuple[str, str]

class mmdet.evaluation.metrics.CocoVideoMetric(ann_file: Optional[str] = None, metric: Union[str, List[str]] = 'bbox', classwise: bool = False, proposal_nums: Sequence[int] = (100, 300, 1000), iou_thrs: Optional[Union[float, Sequence[float]]] = None, metric_items: Optional[Sequence[str]] = None, format_only: bool = False, outfile_prefix: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, collect_device: str = 'cpu', prefix: Optional[str] = None, sort_categories: bool = False, use_mp_eval: bool = False)

COCO evaluation metric.

Evaluate AR, AP, and mAP for detection tasks including proposal/box detection and instance segmentation. Please refer to https://cocodataset.org/#detection-eval for more details.

evaluate(size: int = 1) → dict

Evaluate the model performance of the whole dataset after processing all batches.

Parameters

size (int) – Length of the entire validation dataset.

Returns

Evaluation metrics dict on the val dataset. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions.

The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

class mmdet.evaluation.metrics.CrowdHumanMetric(ann_file: str, metric: Union[str, List[str]] = ['AP', 'MR', 'JI'], format_only: bool = False, outfile_prefix: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, collect_device: str = 'cpu', prefix: Optional[str] = None, eval_mode: int = 0, iou_thres: float = 0.5, compare_matching_method: Optional[str] = None, mr_ref: str = 'CALTECH_-2', num_ji_process: int = 10)

CrowdHuman evaluation metric.

Evaluate Average Precision (AP), Miss Rate (MR) and Jaccard Index (JI) for detection tasks.

Parameters

• ann_file (str) – Path to the annotation file.

• metric (str | List[str]) – Metrics to be evaluated. Valid metrics include ‘AP’, ‘MR’ and ‘JI’. Defaults to ‘AP’.

• format_only (bool) – Format the output results without perform evaluation. It is useful when you want to format the result to a specific format and submit it to the test server. Defaults to False.

• outfile_prefix (str, optional) – The prefix of json files. It includes the file path and the prefix of filename, e.g., “a/b/prefix”. If not specified, a temp file will be created. Defaults to None.

• file_client_args (dict, optional) – Arguments to instantiate the corresponding backend in mmdet <= 3.0.0rc6. Defaults to None.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

• eval_mode (int) – Select the mode of evaluate. Valid mode include 0(just body box), 1(just head box) and 2(both of them). Defaults to 0.

• iou_thres (float) – IoU threshold. Defaults to 0.5.

• compare_matching_method (str, optional) – Matching method to compare the detection results with the ground_truth when compute ‘AP’ and ‘MR’.Valid method include VOC and None(CALTECH). Default to None.

• mr_ref (str) – Different parameter selection to calculate MR. Valid ref include CALTECH_-2 and CALTECH_-4. Defaults to CALTECH_-2.

• num_ji_process (int) – The number of processes to evaluation JI. Defaults to 10.

compare(samples)

Match the detection results with the ground_truth.

Parameters

samples (dict[Image]) – The detection result packaged by Image.

Returns

Matching result. a list of tuples (dtbox, label, imgID) in the descending sort of dtbox.score.

Return type

score_list(list[tuple[ndarray, int, str]])

compute_ji_matching(dt_boxes, gt_boxes)

Match the annotation box for each detection box.

Parameters

• dt_boxes (ndarray) – Detection boxes.

• gt_boxes (ndarray) – Ground_truth boxes.

Returns

Match result.

Return type

matches_(list[tuple[int, int]])

compute_ji_with_ignore(result_queue, dt_result, score_thr)

Compute JI with ignore.

Parameters

• result_queue (Queue) – The Queue for save compute result when multi_process.

• dt_result (dict[Image]) – Detection result packaged by Image.

• score_thr (float) – The threshold of detection score.

Returns

compute result.

Return type

dict

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

eval_results(Dict[str, float])

static eval_ap(score_list, gt_num, img_num)

Evaluate by average precision.

Parameters

• score_list (list[tuple[ndarray, int, str]]) – Matching result. a list of tuples (dtbox, label, imgID) in the descending sort of dtbox.score.

• gt_num (int) – The number of gt boxes in the entire dataset.

• img_num (int) – The number of images in the entire dataset.

Returns

result of average precision.

Return type

ap(float)

eval_ji(samples)

Evaluate by JI using multi_process.

Parameters

samples (Dict[str, Image]) – The detection result packaged by Image.

Returns

result of jaccard index.

Return type

ji(float)

eval_mr(score_list, gt_num, img_num)

Evaluate by Caltech-style log-average miss rate.

Parameters

• score_list (list[tuple[ndarray, int, str]]) – Matching result. a list of tuples (dtbox, label, imgID) in the descending sort of dtbox.score.

• gt_num (int) – The number of gt boxes in the entire dataset.

• img_num (int) – The number of image in the entire dataset.

Returns

result of miss rate.

Return type

mr(float)

static gather(results)

Integrate test results.

get_ignores(dt_boxes, gt_boxes)

Get the number of ignore bboxes.

load_eval_samples(result_file)

Load data from annotations file and detection results.

Parameters

result_file (str) – The file path of the saved detection results.

Returns

The detection result packaged by Image

Return type

Dict[Image]

process(data_batch: Sequence[dict], data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

static results2json(results: Sequence[dict], outfile_prefix: str) → str

Dump the detection results to a json file.

class mmdet.evaluation.metrics.DODCocoMetric(ann_file: Optional[str] = None, collect_device: str = 'cpu', outfile_prefix: Optional[str] = None, backend_args: Optional[dict] = None, prefix: Optional[str] = None)

compute_metrics(results: list) → dict

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (Any) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of outputs from the model.

results2json(results: Sequence[dict]) → list

Dump the detection results to a COCO style json file.

There are 3 types of results: proposals, bbox predictions, mask predictions, and they have different data types. This method will automatically recognize the type, and dump them to json files.

Parameters

results (Sequence[dict]) – Testing results of the dataset.

Returns

Possible keys are “bbox”, “segm”, “proposal”, and values are corresponding filenames.

Return type

dict

xyxy2xywh(bbox: ndarray) → list

Convert xyxy style bounding boxes to xywh style for COCO evaluation.

Parameters

bbox (numpy.ndarray) – The bounding boxes, shape (4, ), in xyxy order.

Returns

The converted bounding boxes, in xywh order.

Return type

list[float]

class mmdet.evaluation.metrics.DumpDetResults(out_file_path: str, collect_device: str = 'cpu', collect_dir: Optional[str] = None)

Dump model predictions to a pickle file for offline evaluation.

Different from DumpResults in MMEngine, it compresses instance segmentation masks into RLE format.

Parameters

• out_file_path (str) – Path of the dumped file. Must end with ‘.pkl’ or ‘.pickle’.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

process(data_batch: dict, data_samples: Sequence[dict]) → None

Transfer tensors in predictions to CPU.

class mmdet.evaluation.metrics.DumpODVGResults(outfile_path, img_prefix: str, score_thr: float = 0.1, collect_device: str = 'cpu', nms_thr: float = 0.5, prefix: Optional[str] = None)

compute_metrics(results: list) → dict

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: Any, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (Any) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of outputs from the model.

class mmdet.evaluation.metrics.DumpProposals(output_dir: str = '', proposals_file: str = 'proposals.pkl', num_max_proposals: Optional[int] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, collect_device: str = 'cpu', prefix: Optional[str] = None)

Dump proposals pseudo metric.

Parameters

• output_dir (str) – The root directory for proposals_file. Defaults to ‘’.

• proposals_file (str) – Proposals file path. Defaults to ‘proposals.pkl’.

• num_max_proposals (int, optional) – Maximum number of proposals to dump. If not specified, all proposals will be dumped.

• file_client_args (dict, optional) – Arguments to instantiate the corresponding backend in mmdet <= 3.0.0rc6. Defaults to None.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

compute_metrics(results: list) → dict

Dump the processed results.

Parameters

results (list) – The processed results of each batch.

Returns

An empty dict.

Return type

dict

process(data_batch: Sequence[dict], data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

class mmdet.evaluation.metrics.FasterCocoMetric(ann_file: Optional[str] = None, metric: Union[str, List[str]] = 'bbox', classwise: bool = False, proposal_nums: Sequence[int] = (100, 300, 1000), iou_thrs: Optional[Union[float, Sequence[float]]] = None, metric_items: Optional[Sequence[str]] = None, format_only: bool = False, outfile_prefix: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, collect_device: str = 'cpu', prefix: Optional[str] = None, sort_categories: bool = False, use_mp_eval: bool = False)

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

Dict[str, float]

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

class mmdet.evaluation.metrics.Flickr30kMetric(topk: Sequence[int] = (1, 5, 10, -1), iou_thrs: float = 0.5, merge_boxes: bool = False, collect_device: str = 'cpu', prefix: Optional[str] = None)

Phrase Grounding Metric.

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

Dict[str, float]

merge_boxes(boxes: List[List[int]]) → List[List[int]]

Return the boxes corresponding to the smallest enclosing box containing all the provided boxes The boxes are expected in [x1, y1, x2, y2] format.

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions.

The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed. :param data_batch: A batch of data from the dataloader. :type data_batch: dict :param data_samples: A batch of data samples that

contain annotations and predictions.

class mmdet.evaluation.metrics.LVISMetric(ann_file: Optional[str] = None, metric: Union[str, List[str]] = 'bbox', classwise: bool = False, proposal_nums: Sequence[int] = (100, 300, 1000), iou_thrs: Optional[Union[float, Sequence[float]]] = None, metric_items: Optional[Sequence[str]] = None, format_only: bool = False, outfile_prefix: Optional[str] = None, collect_device: str = 'cpu', prefix: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None)

LVIS evaluation metric.

Parameters

• ann_file (str, optional) – Path to the coco format annotation file. If not specified, ground truth annotations from the dataset will be converted to coco format. Defaults to None.

• metric (str | List[str]) – Metrics to be evaluated. Valid metrics include ‘bbox’, ‘segm’, ‘proposal’, and ‘proposal_fast’. Defaults to ‘bbox’.

• classwise (bool) – Whether to evaluate the metric class-wise. Defaults to False.

• proposal_nums (Sequence[int]) – Numbers of proposals to be evaluated. Defaults to (100, 300, 1000).

• iou_thrs (float | List[float], optional) – IoU threshold to compute AP and AR. If not specified, IoUs from 0.5 to 0.95 will be used. Defaults to None.

• metric_items (List[str], optional) – Metric result names to be recorded in the evaluation result. Defaults to None.

• format_only (bool) – Format the output results without perform evaluation. It is useful when you want to format the result to a specific format and submit it to the test server. Defaults to False.

• outfile_prefix (str, optional) – The prefix of json files. It includes the file path and the prefix of filename, e.g., “a/b/prefix”. If not specified, a temp file will be created. Defaults to None.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

• file_client_args (dict, optional) – Arguments to instantiate the corresponding backend in mmdet <= 3.0.0rc6. Defaults to None.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

Dict[str, float]

fast_eval_recall(results: List[dict], proposal_nums: Sequence[int], iou_thrs: Sequence[float], logger: Optional[MMLogger] = None) → ndarray

Evaluate proposal recall with LVIS’s fast_eval_recall.

Parameters

• results (List[dict]) – Results of the dataset.

• proposal_nums (Sequence[int]) – Proposal numbers used for evaluation.

• iou_thrs (Sequence[float]) – IoU thresholds used for evaluation.

• logger (MMLogger, optional) – Logger used for logging the recall summary.

Returns

Averaged recall results.

Return type

np.ndarray

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

class mmdet.evaluation.metrics.MOTChallengeMetric(metric: Union[str, List[str]] = ['HOTA', 'CLEAR', 'Identity'], outfile_prefix: Optional[str] = None, track_iou_thr: float = 0.5, benchmark: str = 'MOT17', format_only: bool = False, use_postprocess: bool = False, postprocess_tracklet_cfg: Optional[List[dict]] = [], collect_device: str = 'cpu', prefix: Optional[str] = None)

Evaluation metrics for MOT Challenge.

Parameters

• metric (str | list[str]) – Metrics to be evaluated. Options are ‘HOTA’, ‘CLEAR’, ‘Identity’. Defaults to [‘HOTA’, ‘CLEAR’, ‘Identity’].

• outfile_prefix (str, optional) – Path to save the formatted results. Defaults to None.

• track_iou_thr (float) – IoU threshold for tracking evaluation. Defaults to 0.5.

• benchmark (str) – Benchmark to be evaluated. Defaults to ‘MOT17’.

• format_only (bool) – If True, only formatting the results to the official format and not performing evaluation. Defaults to False.

• postprocess_tracklet_cfg (List[dict], optional) –

  configs for tracklets postprocessing methods. InterpolateTracklets is supported. Defaults to [] - InterpolateTracklets:

  • min_num_frames (int, optional): The minimum length of a

    track that will be interpolated. Defaults to 5.

  • max_num_frames (int, optional): The maximum disconnected

    length in a track. Defaults to 20.

  • use_gsi (bool, optional): Whether to use the GSI (Gaussian-

    smoothed interpolation) method. Defaults to False.

  • smooth_tau (int, optional): smoothing parameter in GSI.

    Defaults to 10.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Default: None

Returns:

compute_metrics(results: Optional[list] = None) → dict

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch. Defaults to None.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

evaluate(size: int = 1) → dict

Evaluate the model performance of the whole dataset after processing all batches.

Parameters

size (int) – Length of the entire validation dataset. Defaults to None.

Returns

Evaluation metrics dict on the val dataset. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

get_dataset_cfg(gt_folder: str, tracker_folder: str)

Get default configs for trackeval.datasets.MotChallenge2DBox.

Parameters

• gt_folder (str) – the name of the GT folder

• tracker_folder (str) – the name of the tracker folder

Returns

Dataset Configs for MotChallenge2DBox.

class mmdet.evaluation.metrics.OVCocoMetric(ann_file: Optional[str] = None, metric: Union[str, List[str]] = 'bbox', classwise: bool = False, proposal_nums: Sequence[int] = (100, 300, 1000), iou_thrs: Optional[Union[float, Sequence[float]]] = None, metric_items: Optional[Sequence[str]] = None, format_only: bool = False, outfile_prefix: Optional[str] = None, file_client_args: Optional[dict] = None, backend_args: Optional[dict] = None, collect_device: str = 'cpu', prefix: Optional[str] = None, sort_categories: bool = False, use_mp_eval: bool = False)

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

Dict[str, float]

class mmdet.evaluation.metrics.OpenImagesMetric(iou_thrs: Union[float, List[float]] = 0.5, ioa_thrs: Union[float, List[float]] = 0.5, scale_ranges: Optional[List[tuple]] = None, use_group_of: bool = True, get_supercategory: bool = True, filter_labels: bool = True, collect_device: str = 'cpu', prefix: Optional[str] = None)

OpenImages evaluation metric.

Evaluate detection mAP for OpenImages. Please refer to https://storage.googleapis.com/openimages/web/evaluation.html for more details.

Parameters

• iou_thrs (float or List[float]) – IoU threshold. Defaults to 0.5.

• ioa_thrs (float or List[float]) – IoA threshold. Defaults to 0.5.

• scale_ranges (List[tuple], optional) – Scale ranges for evaluating mAP. If not specified, all bounding boxes would be included in evaluation. Defaults to None

• use_group_of (bool) – Whether consider group of groud truth bboxes during evaluating. Defaults to True.

• get_supercategory (bool) – Whether to get parent class of the current class. Default: True.

• filter_labels (bool) – Whether filter unannotated classes. Default: True.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

compute_metrics(results: list) → dict

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

class mmdet.evaluation.metrics.ReIDMetrics(metric: Union[str, Sequence[str]] = 'mAP', metric_options: Optional[dict] = None, collect_device: str = 'cpu', prefix: Optional[str] = None)

MAP and CMC evaluation metrics for the ReID task.

Parameters

• metric (str | list[str]) – Metrics to be evaluated. Default value is mAP.

• metric_options – (dict, optional): Options for calculating metrics. Allowed keys are ‘rank_list’ and ‘max_rank’. Defaults to None.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Default: None

compute_metrics(results: list) → dict

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions.

The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

class mmdet.evaluation.metrics.RefExpMetric(ann_file: Optional[str] = None, metric: str = 'bbox', topk=(1, 5, 10), iou_thrs: float = 0.5, **kwargs)

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (Any) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of outputs from the model.

class mmdet.evaluation.metrics.RefSegMetric(metric: Sequence = ('cIoU', 'mIoU'), **kwargs)

Referring Expression Segmentation Metric.

compute_metrics(results: list) → dict

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data and data_samples.

The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of outputs from the model.

class mmdet.evaluation.metrics.SemSegMetric(iou_metrics: Sequence[str] = ['mIoU'], beta: int = 1, collect_device: str = 'cpu', output_dir: Optional[str] = None, format_only: bool = False, backend_args: Optional[dict] = None, prefix: Optional[str] = None)

MIoU evaluation metric.

Parameters

• iou_metrics (list[str] | str) – Metrics to be calculated, the options includes ‘mIoU’, ‘mDice’ and ‘mFscore’.

• beta (int) – Determines the weight of recall in the combined score. Default: 1.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• output_dir (str) – The directory for output prediction. Defaults to None.

• format_only (bool) – Only format result for results commit without perform evaluation. It is useful when you want to save the result to a specific format and submit it to the test server. Defaults to False.

• backend_args (dict, optional) – Arguments to instantiate the corresponding backend. Defaults to None.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of

the metrics, and the values are corresponding results. The key mainly includes aAcc, mIoU, mAcc, mDice, mFscore, mPrecision, mRecall.

Return type

Dict[str, float]

get_return_metrics(results: list) → dict

Calculate evaluation metrics.

Parameters

results (list) – The processed results of each batch.

Returns

per category evaluation metrics,

shape (num_classes, ).

Return type

Dict[str, np.ndarray]

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data and data_samples.

The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of outputs from the model.

class mmdet.evaluation.metrics.VOCMetric(iou_thrs: Union[float, List[float]] = 0.5, scale_ranges: Optional[List[tuple]] = None, metric: Union[str, List[str]] = 'mAP', proposal_nums: Sequence[int] = (100, 300, 1000), eval_mode: str = '11points', collect_device: str = 'cpu', prefix: Optional[str] = None)

Pascal VOC evaluation metric.

Parameters

• iou_thrs (float or List[float]) – IoU threshold. Defaults to 0.5.

• scale_ranges (List[tuple], optional) – Scale ranges for evaluating mAP. If not specified, all bounding boxes would be included in evaluation. Defaults to None.

• metric (str | list[str]) –

  Metrics to be evaluated. Options are ‘mAP’, ‘recall’. If is list, the first setting in the list will

  be used to evaluate metric.

• proposal_nums (Sequence[int]) – Proposal number used for evaluating recalls, such as recall@100, recall@1000. Default: (100, 300, 1000).

• eval_mode (str) – ‘area’ or ‘11points’, ‘area’ means calculating the area under precision-recall curve, ‘11points’ means calculating the average precision of recalls at [0, 0.1, …, 1]. The PASCAL VOC2007 defaults to use ‘11points’, while PASCAL VOC2012 defaults to use ‘area’.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonymous metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Defaults to None.

compute_metrics(results: list) → dict

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (dict) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of data samples that contain annotations and predictions.

class mmdet.evaluation.metrics.YouTubeVISMetric(metric: Union[str, List[str]] = 'youtube_vis_ap', metric_items: Optional[Sequence[str]] = None, outfile_prefix: Optional[str] = None, collect_device: str = 'cpu', prefix: Optional[str] = None, format_only: bool = False)

MAP evaluation metrics for the VIS task.

Parameters

• metric (str | list[str]) – Metrics to be evaluated. Default value is youtube_vis_ap.

• metric_items (List[str], optional) – Metric result names to be recorded in the evaluation result. Defaults to None.

• outfile_prefix (str | None) – The prefix of json files. It includes the file path and the prefix of filename, e.g., “a/b/prefix”. If not specified, a temp file will be created. Defaults to None.

• collect_device (str) – Device name used for collecting results from different ranks during distributed training. Must be ‘cpu’ or ‘gpu’. Defaults to ‘cpu’.

• prefix (str, optional) – The prefix that will be added in the metric names to disambiguate homonyms metrics of different evaluators. If prefix is not provided in the argument, self.default_prefix will be used instead. Default: None

• format_only (bool) – If True, only formatting the results to the official format and not performing evaluation. Defaults to False.

compute_metrics(results: List) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (List) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

Dict[str, float]

evaluate(size: int) → dict

Evaluate the model performance of the whole dataset after processing all batches.

Parameters

size (int) – Length of the entire validation dataset.

Returns

Evaluation metrics dict on the val dataset. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

format_gts(gts: Tuple[List]) → dict

Gather all ground-truth from self.results.

format_preds(preds: Tuple[List]) → List

Gather all predictions from self.results.

save_pred_results(pred_results: List) → None

Save the results to a zip file (standard format for YouTube-VIS Challenge).

Parameters

pred_results (list) – Testing results of the dataset.

class mmdet.evaluation.metrics.gRefCOCOMetric(ann_file: Optional[str] = None, metric: str = 'bbox', iou_thrs: float = 0.5, thresh_score: float = 0.7, thresh_f1: float = 1.0, **kwargs)

compute_metrics(results: list) → Dict[str, float]

Compute the metrics from processed results.

Parameters

results (list) – The processed results of each batch.

Returns

The computed metrics. The keys are the names of the metrics, and the values are corresponding results.

Return type

dict

process(data_batch: dict, data_samples: Sequence[dict]) → None

Process one batch of data samples and predictions. The processed results should be stored in self.results, which will be used to compute the metrics when all batches have been processed.

Parameters

• data_batch (Any) – A batch of data from the dataloader.

• data_samples (Sequence[dict]) – A batch of outputs from the model.

mmdet.models

backbones

class mmdet.models.backbones.CSPDarknet(arch='P5', deepen_factor=1.0, widen_factor=1.0, out_indices=(2, 3, 4), frozen_stages=-1, use_depthwise=False, arch_ovewrite=None, spp_kernal_sizes=(5, 9, 13), conv_cfg=None, norm_cfg={'eps': 0.001, 'momentum': 0.03, 'type': 'BN'}, act_cfg={'type': 'Swish'}, norm_eval=False, init_cfg={'a': 2.23606797749979, 'distribution': 'uniform', 'layer': 'Conv2d', 'mode': 'fan_in', 'nonlinearity': 'leaky_relu', 'type': 'Kaiming'})

CSP-Darknet backbone used in YOLOv5 and YOLOX.

Parameters

• arch (str) – Architecture of CSP-Darknet, from {P5, P6}. Default: P5.

• deepen_factor (float) – Depth multiplier, multiply number of blocks in CSP layer by this amount. Default: 1.0.

• widen_factor (float) – Width multiplier, multiply number of channels in each layer by this amount. Default: 1.0.

• out_indices (Sequence[int]) – Output from which stages. Default: (2, 3, 4).

• frozen_stages (int) – Stages to be frozen (stop grad and set eval mode). -1 means not freezing any parameters. Default: -1.

• use_depthwise (bool) – Whether to use depthwise separable convolution. Default: False.

• arch_ovewrite (list) – Overwrite default arch settings. Default: None.

• spp_kernal_sizes – (tuple[int]): Sequential of kernel sizes of SPP layers. Default: (5, 9, 13).

• conv_cfg (dict) – Config dict for convolution layer. Default: None.

• norm_cfg (dict) – Dictionary to construct and config norm layer. Default: dict(type=’BN’, requires_grad=True).

• act_cfg (dict) – Config dict for activation layer. Default: dict(type=’LeakyReLU’, negative_slope=0.1).

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None.

Example

>>> from mmdet.models import CSPDarknet
>>> import torch
>>> self = CSPDarknet(depth=53)
>>> self.eval()
>>> inputs = torch.rand(1, 3, 416, 416)
>>> level_outputs = self.forward(inputs)
>>> for level_out in level_outputs:
...     print(tuple(level_out.shape))
...
(1, 256, 52, 52)
(1, 512, 26, 26)
(1, 1024, 13, 13)

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

train(mode=True)

Set the module in training mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Parameters

mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.

Returns

self

Return type

Module

class mmdet.models.backbones.CSPNeXt(arch: str = 'P5', deepen_factor: float = 1.0, widen_factor: float = 1.0, out_indices: Sequence[int] = (2, 3, 4), frozen_stages: int = -1, use_depthwise: bool = False, expand_ratio: float = 0.5, arch_ovewrite: Optional[dict] = None, spp_kernel_sizes: Sequence[int] = (5, 9, 13), channel_attention: bool = True, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'eps': 0.001, 'momentum': 0.03, 'type': 'BN'}, act_cfg: Union[ConfigDict, dict] = {'type': 'SiLU'}, norm_eval: bool = False, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = {'a': 2.23606797749979, 'distribution': 'uniform', 'layer': 'Conv2d', 'mode': 'fan_in', 'nonlinearity': 'leaky_relu', 'type': 'Kaiming'})

CSPNeXt backbone used in RTMDet.

Parameters

• arch (str) – Architecture of CSPNeXt, from {P5, P6}. Defaults to P5.

• expand_ratio (float) – Ratio to adjust the number of channels of the hidden layer. Defaults to 0.5.

• deepen_factor (float) – Depth multiplier, multiply number of blocks in CSP layer by this amount. Defaults to 1.0.

• widen_factor (float) – Width multiplier, multiply number of channels in each layer by this amount. Defaults to 1.0.

• out_indices (Sequence[int]) – Output from which stages. Defaults to (2, 3, 4).

• frozen_stages (int) – Stages to be frozen (stop grad and set eval mode). -1 means not freezing any parameters. Defaults to -1.

• use_depthwise (bool) – Whether to use depthwise separable convolution. Defaults to False.

• arch_ovewrite (list) – Overwrite default arch settings. Defaults to None.

• spp_kernel_sizes – (tuple[int]): Sequential of kernel sizes of SPP layers. Defaults to (5, 9, 13).

• channel_attention (bool) – Whether to add channel attention in each stage. Defaults to True.

• conv_cfg (ConfigDict or dict, optional) – Config dict for convolution layer. Defaults to None.

• norm_cfg (ConfigDict or dict) – Dictionary to construct and config norm layer. Defaults to dict(type=’BN’, requires_grad=True).

• act_cfg (ConfigDict or dict) – Config dict for activation layer. Defaults to dict(type=’SiLU’).

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only.

:param init_cfg (ConfigDict or dict or list[dict] or: list[ConfigDict]): Initialization config dict.

forward(x: Tuple[Tensor, ...]) → Tuple[Tensor, ...]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

train(mode=True) → None

Set the module in training mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Parameters

mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.

Returns

self

Return type

Module

class mmdet.models.backbones.Darknet(depth=53, out_indices=(3, 4, 5), frozen_stages=-1, conv_cfg=None, norm_cfg={'requires_grad': True, 'type': 'BN'}, act_cfg={'negative_slope': 0.1, 'type': 'LeakyReLU'}, norm_eval=True, pretrained=None, init_cfg=None)

Darknet backbone.

Parameters

• depth (int) – Depth of Darknet. Currently only support 53.

• out_indices (Sequence[int]) – Output from which stages.

• frozen_stages (int) – Stages to be frozen (stop grad and set eval mode). -1 means not freezing any parameters. Default: -1.

• conv_cfg (dict) – Config dict for convolution layer. Default: None.

• norm_cfg (dict) – Dictionary to construct and config norm layer. Default: dict(type=’BN’, requires_grad=True)

• act_cfg (dict) – Config dict for activation layer. Default: dict(type=’LeakyReLU’, negative_slope=0.1).

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only.

• pretrained (str, optional) – model pretrained path. Default: None

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

Example

>>> from mmdet.models import Darknet
>>> import torch
>>> self = Darknet(depth=53)
>>> self.eval()
>>> inputs = torch.rand(1, 3, 416, 416)
>>> level_outputs = self.forward(inputs)
>>> for level_out in level_outputs:
...     print(tuple(level_out.shape))
...
(1, 256, 52, 52)
(1, 512, 26, 26)
(1, 1024, 13, 13)

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

static make_conv_res_block(in_channels, out_channels, res_repeat, conv_cfg=None, norm_cfg={'requires_grad': True, 'type': 'BN'}, act_cfg={'negative_slope': 0.1, 'type': 'LeakyReLU'})

In Darknet backbone, ConvLayer is usually followed by ResBlock. This function will make that. The Conv layers always have 3x3 filters with stride=2. The number of the filters in Conv layer is the same as the out channels of the ResBlock.

Parameters

• in_channels (int) – The number of input channels.

• out_channels (int) – The number of output channels.

• res_repeat (int) – The number of ResBlocks.

• conv_cfg (dict) – Config dict for convolution layer. Default: None.

• norm_cfg (dict) – Dictionary to construct and config norm layer. Default: dict(type=’BN’, requires_grad=True)

• act_cfg (dict) – Config dict for activation layer. Default: dict(type=’LeakyReLU’, negative_slope=0.1).

train(mode=True)

Set the module in training mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Parameters

mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.

Returns

self

Return type

Module

class mmdet.models.backbones.DetectoRS_ResNeXt(groups=1, base_width=4, **kwargs)

ResNeXt backbone for DetectoRS.

Parameters

• groups (int) – The number of groups in ResNeXt.

• base_width (int) – The base width of ResNeXt.

make_res_layer(**kwargs)

Pack all blocks in a stage into a ResLayer for DetectoRS.

class mmdet.models.backbones.DetectoRS_ResNet(sac=None, stage_with_sac=(False, False, False, False), rfp_inplanes=None, output_img=False, pretrained=None, init_cfg=None, **kwargs)

ResNet backbone for DetectoRS.

Parameters

• sac (dict, optional) – Dictionary to construct SAC (Switchable Atrous Convolution). Default: None.

• stage_with_sac (list) – Which stage to use sac. Default: (False, False, False, False).

• rfp_inplanes (int, optional) – The number of channels from RFP. Default: None. If specified, an additional conv layer will be added for rfp_feat. Otherwise, the structure is the same as base class.

• output_img (bool) – If True, the input image will be inserted into the starting position of output. Default: False.

forward(x)

Forward function.

init_weights()

Initialize the weights.

make_res_layer(**kwargs)

Pack all blocks in a stage into a ResLayer for DetectoRS.

rfp_forward(x, rfp_feats)

Forward function for RFP.

class mmdet.models.backbones.EfficientNet(arch='b0', drop_path_rate=0.0, out_indices=(6,), frozen_stages=0, conv_cfg={'type': 'Conv2dAdaptivePadding'}, norm_cfg={'eps': 0.001, 'type': 'BN'}, act_cfg={'type': 'Swish'}, norm_eval=False, with_cp=False, init_cfg=[{'type': 'Kaiming', 'layer': 'Conv2d'}, {'type': 'Constant', 'layer': ['_BatchNorm', 'GroupNorm'], 'val': 1}])

EfficientNet backbone.

Parameters

• arch (str) – Architecture of efficientnet. Defaults to b0.

• out_indices (Sequence[int]) – Output from which stages. Defaults to (6, ).

• frozen_stages (int) – Stages to be frozen (all param fixed). Defaults to 0, which means not freezing any parameters.

• conv_cfg (dict) – Config dict for convolution layer. Defaults to None, which means using conv2d.

• norm_cfg (dict) – Config dict for normalization layer. Defaults to dict(type=’BN’).

• act_cfg (dict) – Config dict for activation layer. Defaults to dict(type=’Swish’).

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only. Defaults to False.

• with_cp (bool) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed. Defaults to False.

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

train(mode=True)

Set the module in training mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Parameters

mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.

Returns

self

Return type

Module

class mmdet.models.backbones.HRNet(extra, in_channels=3, conv_cfg=None, norm_cfg={'type': 'BN'}, norm_eval=True, with_cp=False, zero_init_residual=False, multiscale_output=True, pretrained=None, init_cfg=None)

HRNet backbone.

High-Resolution Representations for Labeling Pixels and Regions arXiv:.

Parameters

• extra (dict) –

  Detailed configuration for each stage of HRNet. There must be 4 stages, the configuration for each stage must have 5 keys:

  • num_modules(int): The number of HRModule in this stage.

  • num_branches(int): The number of branches in the HRModule.

  • block(str): The type of convolution block.

  • num_blocks(tuple): The number of blocks in each branch.

    The length must be equal to num_branches.

  • num_channels(tuple): The number of channels in each branch.

    The length must be equal to num_branches.

• in_channels (int) – Number of input image channels. Default: 3.

• conv_cfg (dict) – Dictionary to construct and config conv layer.

• norm_cfg (dict) – Dictionary to construct and config norm layer.

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only. Default: True.

• with_cp (bool) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed. Default: False.

• zero_init_residual (bool) – Whether to use zero init for last norm layer in resblocks to let them behave as identity. Default: False.

• multiscale_output (bool) – Whether to output multi-level features produced by multiple branches. If False, only the first level feature will be output. Default: True.

• pretrained (str, optional) – Model pretrained path. Default: None.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None.

Example

>>> from mmdet.models import HRNet
>>> import torch
>>> extra = dict(
>>>     stage1=dict(
>>>         num_modules=1,
>>>         num_branches=1,
>>>         block='BOTTLENECK',
>>>         num_blocks=(4, ),
>>>         num_channels=(64, )),
>>>     stage2=dict(
>>>         num_modules=1,
>>>         num_branches=2,
>>>         block='BASIC',
>>>         num_blocks=(4, 4),
>>>         num_channels=(32, 64)),
>>>     stage3=dict(
>>>         num_modules=4,
>>>         num_branches=3,
>>>         block='BASIC',
>>>         num_blocks=(4, 4, 4),
>>>         num_channels=(32, 64, 128)),
>>>     stage4=dict(
>>>         num_modules=3,
>>>         num_branches=4,
>>>         block='BASIC',
>>>         num_blocks=(4, 4, 4, 4),
>>>         num_channels=(32, 64, 128, 256)))
>>> self = HRNet(extra, in_channels=1)
>>> self.eval()
>>> inputs = torch.rand(1, 1, 32, 32)
>>> level_outputs = self.forward(inputs)
>>> for level_out in level_outputs:
...     print(tuple(level_out.shape))
(1, 32, 8, 8)
(1, 64, 4, 4)
(1, 128, 2, 2)
(1, 256, 1, 1)

forward(x)

Forward function.

property norm1

the normalization layer named “norm1”.

Type

nn.Module

property norm2

the normalization layer named “norm2”.

Type

nn.Module

train(mode=True)

Convert the model into training mode will keeping the normalization layer freezed.

class mmdet.models.backbones.HourglassNet(downsample_times: int = 5, num_stacks: int = 2, stage_channels: Sequence = (256, 256, 384, 384, 384, 512), stage_blocks: Sequence = (2, 2, 2, 2, 2, 4), feat_channel: int = 256, norm_cfg: Union[ConfigDict, dict] = {'requires_grad': True, 'type': 'BN'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

HourglassNet backbone.

Stacked Hourglass Networks for Human Pose Estimation. More details can be found in the paper .

Parameters

• downsample_times (int) – Downsample times in a HourglassModule.

• num_stacks (int) – Number of HourglassModule modules stacked, 1 for Hourglass-52, 2 for Hourglass-104.

• stage_channels (Sequence[int]) – Feature channel of each sub-module in a HourglassModule.

• stage_blocks (Sequence[int]) – Number of sub-modules stacked in a HourglassModule.

• feat_channel (int) – Feature channel of conv after a HourglassModule.

• norm_cfg – Dictionary to construct and config norm layer.

Example

>>> from mmdet.models import HourglassNet
>>> import torch
>>> self = HourglassNet()
>>> self.eval()
>>> inputs = torch.rand(1, 3, 511, 511)
>>> level_outputs = self.forward(inputs)
>>> for level_output in level_outputs:
...     print(tuple(level_output.shape))
(1, 256, 128, 128)
(1, 256, 128, 128)

forward(x: Tensor) → List[Tensor]

Forward function.

init_weights() → None

Init module weights.

class mmdet.models.backbones.MobileNetV2(widen_factor=1.0, out_indices=(1, 2, 4, 7), frozen_stages=-1, conv_cfg=None, norm_cfg={'type': 'BN'}, act_cfg={'type': 'ReLU6'}, norm_eval=False, with_cp=False, pretrained=None, init_cfg=None)

MobileNetV2 backbone.

Parameters

• widen_factor (float) – Width multiplier, multiply number of channels in each layer by this amount. Default: 1.0.

• out_indices (Sequence[int], optional) – Output from which stages. Default: (1, 2, 4, 7).

• frozen_stages (int) – Stages to be frozen (all param fixed). Default: -1, which means not freezing any parameters.

• conv_cfg (dict, optional) – Config dict for convolution layer. Default: None, which means using conv2d.

• norm_cfg (dict) – Config dict for normalization layer. Default: dict(type=’BN’).

• act_cfg (dict) – Config dict for activation layer. Default: dict(type=’ReLU6’).

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only. Default: False.

• with_cp (bool) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed. Default: False.

• pretrained (str, optional) – model pretrained path. Default: None

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

forward(x)

Forward function.

make_layer(out_channels, num_blocks, stride, expand_ratio)

Stack InvertedResidual blocks to build a layer for MobileNetV2.

Parameters

• out_channels (int) – out_channels of block.

• num_blocks (int) – number of blocks.

• stride (int) – stride of the first block. Default: 1

• expand_ratio (int) – Expand the number of channels of the hidden layer in InvertedResidual by this ratio. Default: 6.

train(mode=True)

Convert the model into training mode while keep normalization layer frozen.

class mmdet.models.backbones.PyramidVisionTransformer(pretrain_img_size=224, in_channels=3, embed_dims=64, num_stages=4, num_layers=[3, 4, 6, 3], num_heads=[1, 2, 5, 8], patch_sizes=[4, 2, 2, 2], strides=[4, 2, 2, 2], paddings=[0, 0, 0, 0], sr_ratios=[8, 4, 2, 1], out_indices=(0, 1, 2, 3), mlp_ratios=[8, 8, 4, 4], qkv_bias=True, drop_rate=0.0, attn_drop_rate=0.0, drop_path_rate=0.1, use_abs_pos_embed=True, norm_after_stage=False, use_conv_ffn=False, act_cfg={'type': 'GELU'}, norm_cfg={'eps': 1e-06, 'type': 'LN'}, pretrained=None, convert_weights=True, init_cfg=None)

Pyramid Vision Transformer (PVT)

Implementation of Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions.

Parameters

• pretrain_img_size (int | tuple[int]) – The size of input image when pretrain. Defaults: 224.

• in_channels (int) – Number of input channels. Default: 3.

• embed_dims (int) – Embedding dimension. Default: 64.

• num_stags (int) – The num of stages. Default: 4.

• num_layers (Sequence[int]) – The layer number of each transformer encode layer. Default: [3, 4, 6, 3].

• num_heads (Sequence[int]) – The attention heads of each transformer encode layer. Default: [1, 2, 5, 8].

• patch_sizes (Sequence[int]) – The patch_size of each patch embedding. Default: [4, 2, 2, 2].

• strides (Sequence[int]) – The stride of each patch embedding. Default: [4, 2, 2, 2].

• paddings (Sequence[int]) – The padding of each patch embedding. Default: [0, 0, 0, 0].

• sr_ratios (Sequence[int]) – The spatial reduction rate of each transformer encode layer. Default: [8, 4, 2, 1].

• out_indices (Sequence[int] | int) – Output from which stages. Default: (0, 1, 2, 3).

• mlp_ratios (Sequence[int]) – The ratio of the mlp hidden dim to the embedding dim of each transformer encode layer. Default: [8, 8, 4, 4].

• qkv_bias (bool) – Enable bias for qkv if True. Default: True.

• drop_rate (float) – Probability of an element to be zeroed. Default 0.0.

• attn_drop_rate (float) – The drop out rate for attention layer. Default 0.0.

• drop_path_rate (float) – stochastic depth rate. Default 0.1.

• use_abs_pos_embed (bool) – If True, add absolute position embedding to the patch embedding. Defaults: True.

• use_conv_ffn (bool) – If True, use Convolutional FFN to replace FFN. Default: False.

• act_cfg (dict) – The activation config for FFNs. Default: dict(type=’GELU’).

• norm_cfg (dict) – Config dict for normalization layer. Default: dict(type=’LN’).

• pretrained (str, optional) – model pretrained path. Default: None.

• convert_weights (bool) – The flag indicates whether the pre-trained model is from the original repo. We may need to convert some keys to make it compatible. Default: True.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None.

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

init_weights()

Initialize the weights.

class mmdet.models.backbones.PyramidVisionTransformerV2(**kwargs)

Implementation of PVTv2: Improved Baselines with Pyramid Vision Transformer.

class mmdet.models.backbones.RegNet(arch, in_channels=3, stem_channels=32, base_channels=32, strides=(2, 2, 2, 2), dilations=(1, 1, 1, 1), out_indices=(0, 1, 2, 3), style='pytorch', deep_stem=False, avg_down=False, frozen_stages=-1, conv_cfg=None, norm_cfg={'requires_grad': True, 'type': 'BN'}, norm_eval=True, dcn=None, stage_with_dcn=(False, False, False, False), plugins=None, with_cp=False, zero_init_residual=True, pretrained=None, init_cfg=None)

RegNet backbone.

More details can be found in paper .

Parameters

• arch (dict) –

  The parameter of RegNets.

  • w0 (int): initial width

  • wa (float): slope of width

  • wm (float): quantization parameter to quantize the width

  • depth (int): depth of the backbone

  • group_w (int): width of group

  • bot_mul (float): bottleneck ratio, i.e. expansion of bottleneck.

• strides (Sequence[int]) – Strides of the first block of each stage.

• base_channels (int) – Base channels after stem layer.

• in_channels (int) – Number of input image channels. Default: 3.

• dilations (Sequence[int]) – Dilation of each stage.

• out_indices (Sequence[int]) – Output from which stages.

• style (str) – pytorch or caffe. If set to “pytorch”, the stride-two layer is the 3x3 conv layer, otherwise the stride-two layer is the first 1x1 conv layer.

• frozen_stages (int) – Stages to be frozen (all param fixed). -1 means not freezing any parameters.

• norm_cfg (dict) – dictionary to construct and config norm layer.

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only.

• with_cp (bool) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed.

• zero_init_residual (bool) – whether to use zero init for last norm layer in resblocks to let them behave as identity.

• pretrained (str, optional) – model pretrained path. Default: None

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

Example

>>> from mmdet.models import RegNet
>>> import torch
>>> self = RegNet(
        arch=dict(
            w0=88,
            wa=26.31,
            wm=2.25,
            group_w=48,
            depth=25,
            bot_mul=1.0))
>>> self.eval()
>>> inputs = torch.rand(1, 3, 32, 32)
>>> level_outputs = self.forward(inputs)
>>> for level_out in level_outputs:
...     print(tuple(level_out.shape))
(1, 96, 8, 8)
(1, 192, 4, 4)
(1, 432, 2, 2)
(1, 1008, 1, 1)

adjust_width_group(widths, bottleneck_ratio, groups)

Adjusts the compatibility of widths and groups.

Parameters

• widths (list[int]) – Width of each stage.

• bottleneck_ratio (float) – Bottleneck ratio.

• groups (int) – number of groups in each stage

Returns

The adjusted widths and groups of each stage.

Return type

tuple(list)

forward(x)

Forward function.

generate_regnet(initial_width, width_slope, width_parameter, depth, divisor=8)

Generates per block width from RegNet parameters.

Parameters

• initial_width ([int]) – Initial width of the backbone

• width_slope ([float]) – Slope of the quantized linear function

• width_parameter ([int]) – Parameter used to quantize the width.

• depth ([int]) – Depth of the backbone.

• divisor (int, optional) – The divisor of channels. Defaults to 8.

Returns

return a list of widths of each stage and the number of stages

Return type

list, int

get_stages_from_blocks(widths)

Gets widths/stage_blocks of network at each stage.

Parameters

widths (list[int]) – Width in each stage.

Returns

width and depth of each stage

Return type

tuple(list)

static quantize_float(number, divisor)

Converts a float to closest non-zero int divisible by divisor.

Parameters

• number (int) – Original number to be quantized.

• divisor (int) – Divisor used to quantize the number.

Returns

quantized number that is divisible by devisor.

Return type

int

class mmdet.models.backbones.Res2Net(scales=4, base_width=26, style='pytorch', deep_stem=True, avg_down=True, pretrained=None, init_cfg=None, **kwargs)

Res2Net backbone.

Parameters

• scales (int) – Scales used in Res2Net. Default: 4

• base_width (int) – Basic width of each scale. Default: 26

• depth (int) – Depth of res2net, from {50, 101, 152}.

• in_channels (int) – Number of input image channels. Default: 3.

• num_stages (int) – Res2net stages. Default: 4.

• strides (Sequence[int]) – Strides of the first block of each stage.

• dilations (Sequence[int]) – Dilation of each stage.

• out_indices (Sequence[int]) – Output from which stages.

• style (str) – pytorch or caffe. If set to “pytorch”, the stride-two layer is the 3x3 conv layer, otherwise the stride-two layer is the first 1x1 conv layer.

• deep_stem (bool) – Replace 7x7 conv in input stem with 3 3x3 conv

• avg_down (bool) – Use AvgPool instead of stride conv when downsampling in the bottle2neck.

• frozen_stages (int) – Stages to be frozen (stop grad and set eval mode). -1 means not freezing any parameters.

• norm_cfg (dict) – Dictionary to construct and config norm layer.

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only.

• plugins (list[dict]) –

  List of plugins for stages, each dict contains:

  • cfg (dict, required): Cfg dict to build plugin.

  • position (str, required): Position inside block to insert plugin, options are ‘after_conv1’, ‘after_conv2’, ‘after_conv3’.

  • stages (tuple[bool], optional): Stages to apply plugin, length should be same as ‘num_stages’.

• with_cp (bool) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed.

• zero_init_residual (bool) – Whether to use zero init for last norm layer in resblocks to let them behave as identity.

• pretrained (str, optional) – model pretrained path. Default: None

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

Example

>>> from mmdet.models import Res2Net
>>> import torch
>>> self = Res2Net(depth=50, scales=4, base_width=26)
>>> self.eval()
>>> inputs = torch.rand(1, 3, 32, 32)
>>> level_outputs = self.forward(inputs)
>>> for level_out in level_outputs:
...     print(tuple(level_out.shape))
(1, 256, 8, 8)
(1, 512, 4, 4)
(1, 1024, 2, 2)
(1, 2048, 1, 1)

make_res_layer(**kwargs)

Pack all blocks in a stage into a ResLayer.

class mmdet.models.backbones.ResNeSt(groups=1, base_width=4, radix=2, reduction_factor=4, avg_down_stride=True, **kwargs)

ResNeSt backbone.

Parameters

• groups (int) – Number of groups of Bottleneck. Default: 1

• base_width (int) – Base width of Bottleneck. Default: 4

• radix (int) – Radix of SplitAttentionConv2d. Default: 2

• reduction_factor (int) – Reduction factor of inter_channels in SplitAttentionConv2d. Default: 4.

• avg_down_stride (bool) – Whether to use average pool for stride in Bottleneck. Default: True.

• kwargs (dict) – Keyword arguments for ResNet.

make_res_layer(**kwargs)

Pack all blocks in a stage into a ResLayer.

class mmdet.models.backbones.ResNeXt(groups=1, base_width=4, **kwargs)

ResNeXt backbone.

Parameters

• depth (int) – Depth of resnet, from {18, 34, 50, 101, 152}.

• in_channels (int) – Number of input image channels. Default: 3.

• num_stages (int) – Resnet stages. Default: 4.

• groups (int) – Group of resnext.

• base_width (int) – Base width of resnext.

• strides (Sequence[int]) – Strides of the first block of each stage.

• dilations (Sequence[int]) – Dilation of each stage.

• out_indices (Sequence[int]) – Output from which stages.

• style (str) – pytorch or caffe. If set to “pytorch”, the stride-two layer is the 3x3 conv layer, otherwise the stride-two layer is the first 1x1 conv layer.

• frozen_stages (int) – Stages to be frozen (all param fixed). -1 means not freezing any parameters.

• norm_cfg (dict) – dictionary to construct and config norm layer.

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only.

• with_cp (bool) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed.

• zero_init_residual (bool) – whether to use zero init for last norm layer in resblocks to let them behave as identity.

make_res_layer(**kwargs)

Pack all blocks in a stage into a ResLayer

class mmdet.models.backbones.ResNet(depth, in_channels=3, stem_channels=None, base_channels=64, num_stages=4, strides=(1, 2, 2, 2), dilations=(1, 1, 1, 1), out_indices=(0, 1, 2, 3), style='pytorch', deep_stem=False, avg_down=False, frozen_stages=-1, conv_cfg=None, norm_cfg={'requires_grad': True, 'type': 'BN'}, norm_eval=True, dcn=None, stage_with_dcn=(False, False, False, False), plugins=None, with_cp=False, zero_init_residual=True, pretrained=None, init_cfg=None)

ResNet backbone.

Parameters

• depth (int) – Depth of resnet, from {18, 34, 50, 101, 152}.

• stem_channels (int | None) – Number of stem channels. If not specified, it will be the same as base_channels. Default: None.

• base_channels (int) – Number of base channels of res layer. Default: 64.

• in_channels (int) – Number of input image channels. Default: 3.

• num_stages (int) – Resnet stages. Default: 4.

• strides (Sequence[int]) – Strides of the first block of each stage.

• dilations (Sequence[int]) – Dilation of each stage.

• out_indices (Sequence[int]) – Output from which stages.

• style (str) – pytorch or caffe. If set to “pytorch”, the stride-two layer is the 3x3 conv layer, otherwise the stride-two layer is the first 1x1 conv layer.

• deep_stem (bool) – Replace 7x7 conv in input stem with 3 3x3 conv

• avg_down (bool) – Use AvgPool instead of stride conv when downsampling in the bottleneck.

• frozen_stages (int) – Stages to be frozen (stop grad and set eval mode). -1 means not freezing any parameters.

• norm_cfg (dict) – Dictionary to construct and config norm layer.

• norm_eval (bool) – Whether to set norm layers to eval mode, namely, freeze running stats (mean and var). Note: Effect on Batch Norm and its variants only.

• plugins (list[dict]) –

  List of plugins for stages, each dict contains:

  • cfg (dict, required): Cfg dict to build plugin.

  • position (str, required): Position inside block to insert plugin, options are ‘after_conv1’, ‘after_conv2’, ‘after_conv3’.

  • stages (tuple[bool], optional): Stages to apply plugin, length should be same as ‘num_stages’.

• with_cp (bool) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed.

• zero_init_residual (bool) – Whether to use zero init for last norm layer in resblocks to let them behave as identity.

• pretrained (str, optional) – model pretrained path. Default: None

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

Example

>>> from mmdet.models import ResNet
>>> import torch
>>> self = ResNet(depth=18)
>>> self.eval()
>>> inputs = torch.rand(1, 3, 32, 32)
>>> level_outputs = self.forward(inputs)
>>> for level_out in level_outputs:
...     print(tuple(level_out.shape))
(1, 64, 8, 8)
(1, 128, 4, 4)
(1, 256, 2, 2)
(1, 512, 1, 1)

forward(x)

Forward function.

make_res_layer(**kwargs)

Pack all blocks in a stage into a ResLayer.

make_stage_plugins(plugins, stage_idx)

Make plugins for ResNet stage_idx th stage.

Currently we support to insert context_block, empirical_attention_block, nonlocal_block into the backbone like ResNet/ResNeXt. They could be inserted after conv1/conv2/conv3 of Bottleneck.

An example of plugins format could be:

Examples

>>> plugins=[
...     dict(cfg=dict(type='xxx', arg1='xxx'),
...          stages=(False, True, True, True),
...          position='after_conv2'),
...     dict(cfg=dict(type='yyy'),
...          stages=(True, True, True, True),
...          position='after_conv3'),
...     dict(cfg=dict(type='zzz', postfix='1'),
...          stages=(True, True, True, True),
...          position='after_conv3'),
...     dict(cfg=dict(type='zzz', postfix='2'),
...          stages=(True, True, True, True),
...          position='after_conv3')
... ]
>>> self = ResNet(depth=18)
>>> stage_plugins = self.make_stage_plugins(plugins, 0)
>>> assert len(stage_plugins) == 3

Suppose stage_idx=0, the structure of blocks in the stage would be:

conv1-> conv2->conv3->yyy->zzz1->zzz2

Suppose ‘stage_idx=1’, the structure of blocks in the stage would be:

conv1-> conv2->xxx->conv3->yyy->zzz1->zzz2

If stages is missing, the plugin would be applied to all stages.

Parameters

• plugins (list[dict]) – List of plugins cfg to build. The postfix is required if multiple same type plugins are inserted.

• stage_idx (int) – Index of stage to build

Returns

Plugins for current stage

Return type

list[dict]

property norm1

the normalization layer named “norm1”.

Type

nn.Module

train(mode=True)

Convert the model into training mode while keep normalization layer freezed.

class mmdet.models.backbones.ResNetV1d(**kwargs)

ResNetV1d variant described in Bag of Tricks.

Compared with default ResNet(ResNetV1b), ResNetV1d replaces the 7x7 conv in the input stem with three 3x3 convs. And in the downsampling block, a 2x2 avg_pool with stride 2 is added before conv, whose stride is changed to 1.

class mmdet.models.backbones.SSDVGG(depth, with_last_pool=False, ceil_mode=True, out_indices=(3, 4), out_feature_indices=(22, 34), pretrained=None, init_cfg=None, input_size=None, l2_norm_scale=None)

VGG Backbone network for single-shot-detection.

Parameters

• depth (int) – Depth of vgg, from {11, 13, 16, 19}.

• with_last_pool (bool) – Whether to add a pooling layer at the last of the model

• ceil_mode (bool) – When True, will use ceil instead of floor to compute the output shape.

• out_indices (Sequence[int]) – Output from which stages.

• out_feature_indices (Sequence[int]) – Output from which feature map.

• pretrained (str, optional) – model pretrained path. Default: None

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

• input_size (int, optional) – Deprecated argumment. Width and height of input, from {300, 512}.

• l2_norm_scale (float, optional) – Deprecated argumment. L2 normalization layer init scale.

Example

>>> self = SSDVGG(input_size=300, depth=11)
>>> self.eval()
>>> inputs = torch.rand(1, 3, 300, 300)
>>> level_outputs = self.forward(inputs)
>>> for level_out in level_outputs:
...     print(tuple(level_out.shape))
(1, 1024, 19, 19)
(1, 512, 10, 10)
(1, 256, 5, 5)
(1, 256, 3, 3)
(1, 256, 1, 1)

forward(x)

Forward function.

init_weights(pretrained=None)

Initialize the weights.

class mmdet.models.backbones.SwinTransformer(pretrain_img_size=224, in_channels=3, embed_dims=96, patch_size=4, window_size=7, mlp_ratio=4, depths=(2, 2, 6, 2), num_heads=(3, 6, 12, 24), strides=(4, 2, 2, 2), out_indices=(0, 1, 2, 3), qkv_bias=True, qk_scale=None, patch_norm=True, drop_rate=0.0, attn_drop_rate=0.0, drop_path_rate=0.1, use_abs_pos_embed=False, act_cfg={'type': 'GELU'}, norm_cfg={'type': 'LN'}, with_cp=False, pretrained=None, convert_weights=False, frozen_stages=-1, init_cfg=None)

Swin Transformer A PyTorch implement of : Swin Transformer: Hierarchical Vision Transformer using Shifted Windows -

https://arxiv.org/abs/2103.14030

Inspiration from https://github.com/microsoft/Swin-Transformer

Parameters

• pretrain_img_size (int | tuple[int]) – The size of input image when pretrain. Defaults: 224.

• in_channels (int) – The num of input channels. Defaults: 3.

• embed_dims (int) – The feature dimension. Default: 96.

• patch_size (int | tuple[int]) – Patch size. Default: 4.

• window_size (int) – Window size. Default: 7.

• mlp_ratio (int) – Ratio of mlp hidden dim to embedding dim. Default: 4.

• depths (tuple[int]) – Depths of each Swin Transformer stage. Default: (2, 2, 6, 2).

• num_heads (tuple[int]) – Parallel attention heads of each Swin Transformer stage. Default: (3, 6, 12, 24).

• strides (tuple[int]) – The patch merging or patch embedding stride of each Swin Transformer stage. (In swin, we set kernel size equal to stride.) Default: (4, 2, 2, 2).

• out_indices (tuple[int]) – Output from which stages. Default: (0, 1, 2, 3).

• qkv_bias (bool, optional) – If True, add a learnable bias to query, key, value. Default: True

• qk_scale (float | None, optional) – Override default qk scale of head_dim ** -0.5 if set. Default: None.

• patch_norm (bool) – If add a norm layer for patch embed and patch merging. Default: True.

• drop_rate (float) – Dropout rate. Defaults: 0.

• attn_drop_rate (float) – Attention dropout rate. Default: 0.

• drop_path_rate (float) – Stochastic depth rate. Defaults: 0.1.

• use_abs_pos_embed (bool) – If True, add absolute position embedding to the patch embedding. Defaults: False.

• act_cfg (dict) – Config dict for activation layer. Default: dict(type=’GELU’).

• norm_cfg (dict) – Config dict for normalization layer at output of backone. Defaults: dict(type=’LN’).

• with_cp (bool, optional) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed. Default: False.

• pretrained (str, optional) – model pretrained path. Default: None.

• convert_weights (bool) – The flag indicates whether the pre-trained model is from the original repo. We may need to convert some keys to make it compatible. Default: False.

• frozen_stages (int) – Stages to be frozen (stop grad and set eval mode). Default: -1 (-1 means not freezing any parameters).

• init_cfg (dict, optional) – The Config for initialization. Defaults to None.

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

init_weights()

Initialize the weights.

train(mode=True)

Convert the model into training mode while keep layers freezed.

class mmdet.models.backbones.TridentResNet(depth, num_branch, test_branch_idx, trident_dilations, **kwargs)

The stem layer, stage 1 and stage 2 in Trident ResNet are identical to ResNet, while in stage 3, Trident BottleBlock is utilized to replace the normal BottleBlock to yield trident output. Different branch shares the convolution weight but uses different dilations to achieve multi-scale output.

/ stage3(b0) x - stem - stage1 - stage2 - stage3(b1) - output stage3(b2) /

Parameters

• depth (int) – Depth of resnet, from {50, 101, 152}.

• num_branch (int) – Number of branches in TridentNet.

• test_branch_idx (int) – In inference, all 3 branches will be used if test_branch_idx==-1, otherwise only branch with index test_branch_idx will be used.

• trident_dilations (tuple[int]) – Dilations of different trident branch. len(trident_dilations) should be equal to num_branch.

data_preprocessors

class mmdet.models.data_preprocessors.BatchFixedSizePad(size: Tuple[int, int], img_pad_value: int = 0, pad_mask: bool = False, mask_pad_value: int = 0, pad_seg: bool = False, seg_pad_value: int = 255)

Fixed size padding for batch images.

Parameters

• size (Tuple[int, int]) – Fixed padding size. Expected padding shape (h, w). Defaults to None.

• img_pad_value (int) – The padded pixel value for images. Defaults to 0.

• pad_mask (bool) – Whether to pad instance masks. Defaults to False.

• mask_pad_value (int) – The padded pixel value for instance masks. Defaults to 0.

• pad_seg (bool) – Whether to pad semantic segmentation maps. Defaults to False.

• seg_pad_value (int) – The padded pixel value for semantic segmentation maps. Defaults to 255.

forward(inputs: Tensor, data_samples: Optional[List[dict]] = None) → Tuple[Tensor, Optional[List[dict]]]

Pad image, instance masks, segmantic segmentation maps.

class mmdet.models.data_preprocessors.BatchResize(scale: tuple, pad_size_divisor: int = 1, pad_value: Union[float, int] = 0)

Batch resize during training. This implementation is modified from https://github.com/Purkialo/CrowdDet/blob/master/lib/data/CrowdHuman.py.

It provides the data pre-processing as follows: - A batch of all images will pad to a uniform size and stack them into

a torch.Tensor by DetDataPreprocessor.

• BatchFixShapeResize resize all images to the target size.

• Padding images to make sure the size of image can be divisible by pad_size_divisor.

Parameters

• scale (tuple) – Images scales for resizing.

• pad_size_divisor (int) – Image size divisible factor. Defaults to 1.

• pad_value (Number) – The padded pixel value. Defaults to 0.

forward(inputs: Tensor, data_samples: List[DetDataSample]) → Tuple[Tensor, List[DetDataSample]]

Resize a batch of images and bboxes.

get_padded_tensor(tensor: Tensor, pad_value: int) → Tensor

Pad images according to pad_size_divisor.

get_target_size(height: int, width: int) → Tuple[int, int, float]

Get the target size of a batch of images based on data and scale.

class mmdet.models.data_preprocessors.BatchSyncRandomResize(random_size_range: Tuple[int, int], interval: int = 10, size_divisor: int = 32)

Batch random resize which synchronizes the random size across ranks.

Parameters

• random_size_range (tuple) – The multi-scale random range during multi-scale training.

• interval (int) – The iter interval of change image size. Defaults to 10.

• size_divisor (int) – Image size divisible factor. Defaults to 32.

forward(inputs: Tensor, data_samples: List[DetDataSample]) → Tuple[Tensor, List[DetDataSample]]

Resize a batch of images and bboxes to shape self._input_size

class mmdet.models.data_preprocessors.BoxInstDataPreprocessor(*arg, mask_stride: int = 4, pairwise_size: int = 3, pairwise_dilation: int = 2, pairwise_color_thresh: float = 0.3, bottom_pixels_removed: int = 10, **kwargs)

Pseudo mask pre-processor for BoxInst.

Comparing with the mmdet.DetDataPreprocessor,

1. It generates masks using box annotations.

2. It computes the images color similarity in LAB color space.

Parameters

• mask_stride (int) – The mask output stride in boxinst. Defaults to 4.

• pairwise_size (int) – The size of neighborhood for each pixel. Defaults to 3.

• pairwise_dilation (int) – The dilation of neighborhood for each pixel. Defaults to 2.

• pairwise_color_thresh (float) – The thresh of image color similarity. Defaults to 0.3.

• bottom_pixels_removed (int) – The length of removed pixels in bottom. It is caused by the annotation error in coco dataset. Defaults to 10.

forward(data: dict, training: bool = False) → dict

Get pseudo mask labels using color similarity.

get_images_color_similarity(inputs: Tensor, image_masks: Tensor) → Tensor

Compute the image color similarity in LAB color space.

class mmdet.models.data_preprocessors.DetDataPreprocessor(mean: Optional[Sequence[Number]] = None, std: Optional[Sequence[Number]] = None, pad_size_divisor: int = 1, pad_value: Union[float, int] = 0, pad_mask: bool = False, mask_pad_value: int = 0, pad_seg: bool = False, seg_pad_value: int = 255, bgr_to_rgb: bool = False, rgb_to_bgr: bool = False, boxtype2tensor: bool = True, non_blocking: Optional[bool] = False, batch_augments: Optional[List[dict]] = None)

Image pre-processor for detection tasks.

Comparing with the mmengine.ImgDataPreprocessor,

1. It supports batch augmentations.

2. It will additionally append batch_input_shape and pad_shape to data_samples considering the object detection task.

It provides the data pre-processing as follows

• Collate and move data to the target device.

• Pad inputs to the maximum size of current batch with defined pad_value. The padding size can be divisible by a defined pad_size_divisor

• Stack inputs to batch_inputs.

• Convert inputs from bgr to rgb if the shape of input is (3, H, W).

• Normalize image with defined std and mean.

• Do batch augmentations during training.

Parameters

• mean (Sequence[Number], optional) – The pixel mean of R, G, B channels. Defaults to None.

• std (Sequence[Number], optional) – The pixel standard deviation of R, G, B channels. Defaults to None.

• pad_size_divisor (int) – The size of padded image should be divisible by pad_size_divisor. Defaults to 1.

• pad_value (Number) – The padded pixel value. Defaults to 0.

• pad_mask (bool) – Whether to pad instance masks. Defaults to False.

• mask_pad_value (int) – The padded pixel value for instance masks. Defaults to 0.

• pad_seg (bool) – Whether to pad semantic segmentation maps. Defaults to False.

• seg_pad_value (int) – The padded pixel value for semantic segmentation maps. Defaults to 255.

• bgr_to_rgb (bool) – whether to convert image from BGR to RGB. Defaults to False.

• rgb_to_bgr (bool) – whether to convert image from RGB to RGB. Defaults to False.

• boxtype2tensor (bool) – Whether to convert the BaseBoxes type of bboxes data to Tensor type. Defaults to True.

• non_blocking (bool) – Whether block current process when transferring data to device. Defaults to False.

• batch_augments (list[dict], optional) – Batch-level augmentations

forward(data: dict, training: bool = False) → dict

Perform normalization,padding and bgr2rgb conversion based on BaseDataPreprocessor.

Parameters

• data (dict) – Data sampled from dataloader.

• training (bool) – Whether to enable training time augmentation.

Returns

Data in the same format as the model input.

Return type

dict

pad_gt_masks(batch_data_samples: Sequence[DetDataSample]) → None

Pad gt_masks to shape of batch_input_shape.

pad_gt_sem_seg(batch_data_samples: Sequence[DetDataSample]) → None

Pad gt_sem_seg to shape of batch_input_shape.

class mmdet.models.data_preprocessors.MultiBranchDataPreprocessor(data_preprocessor: Union[ConfigDict, dict])

DataPreprocessor wrapper for multi-branch data.

Take semi-supervised object detection as an example, assume that the ratio of labeled data and unlabeled data in a batch is 1:2, sup indicates the branch where the labeled data is augmented, unsup_teacher and unsup_student indicate the branches where the unlabeled data is augmented by different pipeline.

The input format of multi-branch data is shown as below :

The format of multi-branch data after filtering None is shown as below :

In order to reuse DetDataPreprocessor for the data from different branches, the format of multi-branch data grouped by branch is as below :

After preprocessing data from different branches, the multi-branch data needs to be reformatted as:

Parameters

data_preprocessor (ConfigDict or dict) – Config of DetDataPreprocessor to process the input data.

cpu(*args, **kwargs) → Module

Overrides this method to set the device

Returns

The model itself.

Return type

nn.Module

cuda(*args, **kwargs) → Module

Overrides this method to set the device

Returns

The model itself.

Return type

nn.Module

forward(data: dict, training: bool = False) → dict

Perform normalization,padding and bgr2rgb conversion based on BaseDataPreprocessor for multi-branch data.

Parameters

• data (dict) – Data sampled from dataloader.

• training (bool) – Whether to enable training time augmentation.

Returns

• ‘inputs’ (Dict[str, obj:torch.Tensor]): The forward data of

  models from different branches.

• ’data_sample’ (Dict[str, obj:DetDataSample]): The annotation

  info of the sample from different branches.

Return type

dict

to(device: Optional[Union[int, device]], *args, **kwargs) → Module

Overrides this method to set the device

Parameters

device (int or torch.device, optional) – The desired device of the parameters and buffers in this module.

Returns

The model itself.

Return type

nn.Module

class mmdet.models.data_preprocessors.ReIDDataPreprocessor(mean: Optional[Sequence[Number]] = None, std: Optional[Sequence[Number]] = None, pad_size_divisor: int = 1, pad_value: Number = 0, to_rgb: bool = False, to_onehot: bool = False, num_classes: Optional[int] = None, batch_augments: Optional[dict] = None)

Image pre-processor for classification tasks.

Comparing with the mmengine.model.ImgDataPreprocessor,

1. It won’t do normalization if mean is not specified.

2. It does normalization and color space conversion after stacking batch.

3. It supports batch augmentations like mixup and cutmix.

It provides the data pre-processing as follows

• Collate and move data to the target device.

• Pad inputs to the maximum size of current batch with defined pad_value. The padding size can be divisible by a defined pad_size_divisor

• Stack inputs to batch_inputs.

• Convert inputs from bgr to rgb if the shape of input is (3, H, W).

• Normalize image with defined std and mean.

• Do batch augmentations like Mixup and Cutmix during training.

Parameters

• mean (Sequence[Number], optional) – The pixel mean of R, G, B channels. Defaults to None.

• std (Sequence[Number], optional) – The pixel standard deviation of R, G, B channels. Defaults to None.

• pad_size_divisor (int) – The size of padded image should be divisible by pad_size_divisor. Defaults to 1.

• pad_value (Number) – The padded pixel value. Defaults to 0.

• to_rgb (bool) – whether to convert image from BGR to RGB. Defaults to False.

• to_onehot (bool) – Whether to generate one-hot format gt-labels and set to data samples. Defaults to False.

• num_classes (int, optional) – The number of classes. Defaults to None.

• batch_augments (dict, optional) – The batch augmentations settings, including “augments” and “probs”. For more details, see mmpretrain.models.RandomBatchAugment.

forward(data: dict, training: bool = False) → dict

Perform normalization, padding, bgr2rgb conversion and batch augmentation based on BaseDataPreprocessor.

Parameters

• data (dict) – data sampled from dataloader.

• training (bool) – Whether to enable training time augmentation.

Returns

Data in the same format as the model input.

Return type

dict

class mmdet.models.data_preprocessors.TrackDataPreprocessor(mean: Optional[Sequence[Union[float, int]]] = None, std: Optional[Sequence[Union[float, int]]] = None, use_det_processor: bool = False, **kwargs)

Image pre-processor for tracking tasks.

Accepts the data sampled by the dataloader, and preprocesses it into the format of the model input. TrackDataPreprocessor provides the tracking data pre-processing as follows:

• Collate and move data to the target device.

• Pad inputs to the maximum size of current batch with defined pad_value. The padding size can be divisible by a defined pad_size_divisor

• Stack inputs to inputs.

• Convert inputs from bgr to rgb if the shape of input is (1, 3, H, W).

• Normalize image with defined std and mean.

• Do batch augmentations during training.

• Record the information of batch_input_shape and pad_shape.

Args:

mean (Sequence[Number], optional): The pixel mean of R, G, B

channels. Defaults to None.

std (Sequence[Number], optional): The pixel standard deviation of

R, G, B channels. Defaults to None.

pad_size_divisor (int): The size of padded image should be

divisible by pad_size_divisor. Defaults to 1.

pad_value (Number): The padded pixel value. Defaults to 0. pad_mask (bool): Whether to pad instance masks. Defaults to False. mask_pad_value (int): The padded pixel value for instance masks.

Defaults to 0.

bgr_to_rgb (bool): whether to convert image from BGR to RGB.

Defaults to False.

rgb_to_bgr (bool): whether to convert image from RGB to RGB.

Defaults to False.

use_det_processor: (bool): whether to use DetDataPreprocessor

in training phrase. This is mainly for some tracking models fed into one image rather than a group of image in training. Defaults to False.

. boxtype2tensor (bool): Whether to convert the BaseBoxes type of

bboxes data to Tensor type. Defaults to True.

batch_augments (list[dict], optional): Batch-level augmentations

forward(data: dict, training: bool = False) → Dict

Perform normalization,padding and bgr2rgb conversion based on TrackDataPreprocessor.

Parameters

• data (dict) – data sampled from dataloader.

• training (bool) – Whether to enable training time augmentation.

Returns

Data in the same format as the model input.

Return type

Tuple[Dict[str, List[torch.Tensor]], OptSampleList]

pad_track_gt_masks(data_samples: Sequence[TrackDataSample]) → None

Pad gt_masks to shape of batch_input_shape.

dense_heads

class mmdet.models.dense_heads.ATSSHead(num_classes: int, in_channels: int, pred_kernel_size: int = 3, stacked_convs: int = 4, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 32, 'requires_grad': True, 'type': 'GN'}, reg_decoded_bbox: bool = True, loss_centerness: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'atss_cls', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'}, **kwargs)

Detection Head of ATSS.

ATSS head structure is similar with FCOS, however ATSS use anchor boxes and assign label by Adaptive Training Sample Selection instead max-iou.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• pred_kernel_size (int) – Kernel size of nn.Conv2d

• stacked_convs (int) – Number of stacking convs of the head.

• conv_cfg (ConfigDict or dict, optional) – Config dict for convolution layer. Defaults to None.

• norm_cfg (ConfigDict or dict) – Config dict for normalization layer. Defaults to dict(type='GN', num_groups=32, requires_grad=True).

• reg_decoded_bbox (bool) – If true, the regression loss would be applied directly on decoded bounding boxes, converting both the predicted boxes and regression targets to absolute coordinates format. Defaults to False. It should be True when using IoULoss, GIoULoss, or DIoULoss in the bbox head.

• loss_centerness (ConfigDict or dict) – Config of centerness loss. Defaults to dict(type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0).

:param init_cfg (ConfigDict or dict or list[dict] or: list[ConfigDict]): Initialization config dict.

centerness_target(anchors: Tensor, gts: Tensor) → Tensor

Calculate the centerness between anchors and gts.

Only calculate pos centerness targets, otherwise there may be nan.

Parameters

• anchors (Tensor) – Anchors with shape (N, 4), “xyxy” format.

• gts (Tensor) – Ground truth bboxes with shape (N, 4), “xyxy” format.

Returns

Centerness between anchors and gts.

Return type

Tensor

forward(x: Tuple[Tensor]) → Tuple[List[Tensor]]

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually a tuple of classification scores and bbox prediction

cls_scores (list[Tensor]): Classification scores for all scale

levels, each is a 4D-tensor, the channels number is num_anchors * num_classes.

bbox_preds (list[Tensor]): Box energies / deltas for all scale

levels, each is a 4D-tensor, the channels number is num_anchors * 4.

Return type

tuple

forward_single(x: Tensor, scale: Scale) → Sequence[Tensor]

Forward feature of a single scale level.

Parameters

• x (Tensor) – Features of a single scale level.

• ( (scale) – obj: mmcv.cnn.Scale): Learnable scale module to resize the bbox prediction.

Returns

cls_score (Tensor): Cls scores for a single scale level

the channels number is num_anchors * num_classes.

bbox_pred (Tensor): Box energies / deltas for a single scale

level, the channels number is num_anchors * 4.

centerness (Tensor): Centerness for a single scale level, the

channel number is (N, num_anchors * 1, H, W).

Return type

tuple

get_num_level_anchors_inside(num_level_anchors, inside_flags)

Get the number of valid anchors in every level.

get_targets(anchor_list: List[List[Tensor]], valid_flag_list: List[List[Tensor]], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs: bool = True) → tuple

Get targets for ATSS head.

This method is almost the same as AnchorHead.get_targets(). Besides returning the targets as the parent method does, it also returns the anchors as the first element of the returned tuple.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], centernesses: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• centernesses (list[Tensor]) – Centerness for each scale level with shape (N, num_anchors * 1, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_by_feat_single(anchors: Tensor, cls_score: Tensor, bbox_pred: Tensor, centerness: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, avg_factor: float) → dict

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• cls_score (Tensor) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W).

• bbox_pred (Tensor) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• anchors (Tensor) – Box reference for each scale level with shape (N, num_total_anchors, 4).

• labels (Tensor) – Labels of each anchors with shape (N, num_total_anchors).

• label_weights (Tensor) – Label weights of each anchor with shape (N, num_total_anchors)

• bbox_targets (Tensor) – BBox regression targets of each anchor with shape (N, num_total_anchors, 4).

• avg_factor (float) – Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.ATSSVLFusionHead(*args, early_fuse: bool = False, use_checkpoint: bool = False, num_dyhead_blocks: int = 6, lang_model_name: str = 'bert-base-uncased', init_cfg=None, **kwargs)

ATSS head with visual-language fusion module.

Parameters

• early_fuse (bool) – Whether to fuse visual and language features Defaults to False.

• use_checkpoint (bool) – Whether to use checkpoint. Defaults to False.

• num_dyhead_blocks (int) – Number of dynamic head blocks. Defaults to 6.

• lang_model_name (str) – Name of the language model. Defaults to ‘bert-base-uncased’.

centerness_target(anchors: Tensor, gts: Tensor) → Tensor

Calculate the centerness between anchors and gts.

Only calculate pos centerness targets, otherwise there may be nan.

Parameters

• anchors (Tensor) – Anchors with shape (N, 4), “xyxy” format.

• gts (Tensor) – Ground truth bboxes with shape (N, 4), “xyxy” format.

Returns

Centerness between anchors and gts.

Return type

Tensor

forward(visual_feats: Tuple[Tensor], language_feats: dict) → Tuple[Tensor]

Forward function.

loss(visual_feats: Tuple[Tensor], language_feats: dict, batch_data_samples)

Perform forward propagation and loss calculation of the detection head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], centernesses: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• centernesses (list[Tensor]) – Centerness for each scale level with shape (N, num_anchors * 1, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

predict(visual_feats: Tuple[Tensor], language_feats: dict, batch_data_samples, rescale: bool = True)

Perform forward propagation of the detection head and predict detection results on the features of the upstream network.

Parameters

• visual_feats (tuple[Tensor]) – Multi-level visual features from the upstream network, each is a 4D-tensor.

• language_feats (dict) – Language features from the upstream network.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool, optional) – Whether to rescale the results. Defaults to False.

Returns

InstanceData]: Detection results of each image after the post process.

Return type

list[obj

predict_by_feat(cls_logits: List[Tensor], bbox_preds: List[Tensor], score_factors: List[Tensor], batch_img_metas: Optional[List[dict]] = None, batch_token_positive_maps: Optional[List[dict]] = None, cfg: Optional[ConfigDict] = None, rescale: bool = False, with_nms: bool = True) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Note: When score_factors is not None, the cls_scores are usually multiplied by it then obtain the real score used in NMS, such as CenterNess in FCOS, IoU branch in ATSS.

Parameters

• cls_logits (list[Tensor]) – Classification scores for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * 4, H, W).

• score_factors (list[Tensor], optional) – Score factor for all scale level, each is a 4D-tensor, has shape (batch_size, num_priors * 1, H, W). Defaults to None.

• batch_img_metas (list[dict], Optional) – Batch image meta info. Defaults to None.

• batch_token_positive_maps (list[dict], Optional) – Batch token positive map. Defaults to None.

• cfg (ConfigDict, optional) – Test / postprocessing configuration, if None, test_cfg would be used. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

• with_nms (bool) – If True, do nms before return boxes. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.dense_heads.AnchorFreeHead(num_classes: int, in_channels: int, feat_channels: int = 256, stacked_convs: int = 4, strides: Union[Sequence[int], Sequence[Tuple[int, int]]] = (4, 8, 16, 32, 64), dcn_on_last_conv: bool = False, conv_bias: Union[bool, str] = 'auto', loss_cls: Union[ConfigDict, dict] = {'alpha': 0.25, 'gamma': 2.0, 'loss_weight': 1.0, 'type': 'FocalLoss', 'use_sigmoid': True}, loss_bbox: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'IoULoss'}, bbox_coder: Union[ConfigDict, dict] = {'type': 'DistancePointBBoxCoder'}, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'conv_cls', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'})

Anchor-free head (FCOS, Fovea, RepPoints, etc.).

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• feat_channels (int) – Number of hidden channels. Used in child classes.

• stacked_convs (int) – Number of stacking convs of the head.

• strides (Sequence[int] or Sequence[Tuple[int, int]]) – Downsample factor of each feature map.

• dcn_on_last_conv (bool) – If true, use dcn in the last layer of towers. Defaults to False.

• conv_bias (bool or str) – If specified as auto, it will be decided by the norm_cfg. Bias of conv will be set as True if norm_cfg is None, otherwise False. Default: “auto”.

• loss_cls (ConfigDict or dict) – Config of classification loss.

• loss_bbox (ConfigDict or dict) – Config of localization loss.

• bbox_coder (ConfigDict or dict) – Config of bbox coder. Defaults ‘DistancePointBBoxCoder’.

• conv_cfg (ConfigDict or dict, Optional) – Config dict for convolution layer. Defaults to None.

• norm_cfg (ConfigDict or dict, Optional) – Config dict for normalization layer. Defaults to None.

• train_cfg (ConfigDict or dict, Optional) – Training config of anchor-free head.

• test_cfg (ConfigDict or dict, Optional) – Testing config of anchor-free head.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict]) – Initialization config dict.

aug_test(aug_batch_feats: List[Tensor], aug_batch_img_metas: List[List[Tensor]], rescale: bool = False) → List[ndarray]

Test function with test time augmentation.

Parameters

• aug_batch_feats (list[Tensor]) – the outer list indicates test-time augmentations and inner Tensor should have a shape NxCxHxW, which contains features for all images in the batch.

• aug_batch_img_metas (list[list[dict]]) – the outer list indicates test-time augs (multiscale, flip, etc.) and the inner list indicates images in a batch. each dict has image information.

• rescale (bool, optional) – Whether to rescale the results. Defaults to False.

Returns

bbox results of each class

Return type

list[ndarray]

forward(x: Tuple[Tensor]) → Tuple[List[Tensor], List[Tensor]]

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually contain classification scores and bbox predictions.

• cls_scores (list[Tensor]): Box scores for each scale level, each is a 4D-tensor, the channel number is num_points * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

Return type

tuple

forward_single(x: Tensor) → Tuple[Tensor, ...]

Forward features of a single scale level.

Parameters

x (Tensor) – FPN feature maps of the specified stride.

Returns

Scores for each class, bbox predictions, features after classification and regression conv layers, some models needs these features like FCOS.

Return type

tuple

abstract get_targets(points: List[Tensor], batch_gt_instances: List[InstanceData]) → Any

Compute regression, classification and centerness targets for points in multiple images.

Parameters

• points (list[Tensor]) – Points of each fpn level, each has shape (num_points, 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

abstract loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level, each is a 4D-tensor, the channel number is num_points * num_classes.

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

class mmdet.models.dense_heads.AnchorHead(num_classes: int, in_channels: int, feat_channels: int = 256, anchor_generator: Union[ConfigDict, dict] = {'ratios': [0.5, 1.0, 2.0], 'scales': [8, 16, 32], 'strides': [4, 8, 16, 32, 64], 'type': 'AnchorGenerator'}, bbox_coder: Union[ConfigDict, dict] = {'clip_border': True, 'target_means': (0.0, 0.0, 0.0, 0.0), 'target_stds': (1.0, 1.0, 1.0, 1.0), 'type': 'DeltaXYWHBBoxCoder'}, reg_decoded_bbox: bool = False, loss_cls: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_bbox: Union[ConfigDict, dict] = {'beta': 0.1111111111111111, 'loss_weight': 1.0, 'type': 'SmoothL1Loss'}, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = {'layer': 'Conv2d', 'std': 0.01, 'type': 'Normal'})

Anchor-based head (RPN, RetinaNet, SSD, etc.).

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• feat_channels (int) – Number of hidden channels. Used in child classes.

• anchor_generator (dict) – Config dict for anchor generator

• bbox_coder (dict) – Config of bounding box coder.

• reg_decoded_bbox (bool) – If true, the regression loss would be applied directly on decoded bounding boxes, converting both the predicted boxes and regression targets to absolute coordinates format. Default False. It should be True when using IoULoss, GIoULoss, or DIoULoss in the bbox head.

• loss_cls (dict) – Config of classification loss.

• loss_bbox (dict) – Config of localization loss.

• train_cfg (dict) – Training config of anchor head.

• test_cfg (dict) – Testing config of anchor head.

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(x: Tuple[Tensor]) → Tuple[List[Tensor]]

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of classification scores and bbox prediction.

• cls_scores (list[Tensor]): Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

Return type

tuple

forward_single(x: Tensor) → Tuple[Tensor, Tensor]

Forward feature of a single scale level.

Parameters

x (Tensor) – Features of a single scale level.

Returns

cls_score (Tensor): Cls scores for a single scale level the channels number is num_base_priors * num_classes. bbox_pred (Tensor): Box energies / deltas for a single scale level, the channels number is num_base_priors * 4.

Return type

tuple

get_anchors(featmap_sizes: List[tuple], batch_img_metas: List[dict], device: Union[device, str] = 'cuda') → Tuple[List[List[Tensor]], List[List[Tensor]]]

Get anchors according to feature map sizes.

Parameters

• featmap_sizes (list[tuple]) – Multi-level feature map sizes.

• batch_img_metas (list[dict]) – Image meta info.

• device (torch.device | str) – Device for returned tensors. Defaults to cuda.

Returns

• anchor_list (list[list[Tensor]]): Anchors of each image.

• valid_flag_list (list[list[Tensor]]): Valid flags of each image.

Return type

tuple

get_targets(anchor_list: List[List[Tensor]], valid_flag_list: List[List[Tensor]], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs: bool = True, return_sampling_results: bool = False) → tuple

Compute regression and classification targets for anchors in multiple images.

Parameters

• anchor_list (list[list[Tensor]]) – Multi level anchors of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_anchors, 4).

• valid_flag_list (list[list[Tensor]]) – Multi level valid flags of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_anchors, )

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• unmap_outputs (bool) – Whether to map outputs back to the original set of anchors. Defaults to True.

• return_sampling_results (bool) – Whether to return the sampling results. Defaults to False.

Returns

Usually returns a tuple containing learning targets.

• labels_list (list[Tensor]): Labels of each level.

• label_weights_list (list[Tensor]): Label weights of each level.

• bbox_targets_list (list[Tensor]): BBox targets of each level.

• bbox_weights_list (list[Tensor]): BBox weights of each level.

• avg_factor (int): Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

additional_returns: This function enables user-defined returns from

self._get_targets_single. These returns are currently refined to properties at each feature map (i.e. having HxW dimension). The results will be concatenated after the end

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level has shape (N, num_anchors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict

loss_by_feat_single(cls_score: Tensor, bbox_pred: Tensor, anchors: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, bbox_weights: Tensor, avg_factor: int) → tuple

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• cls_score (Tensor) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W).

• bbox_pred (Tensor) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• anchors (Tensor) – Box reference for each scale level with shape (N, num_total_anchors, 4).

• labels (Tensor) – Labels of each anchors with shape (N, num_total_anchors).

• label_weights (Tensor) – Label weights of each anchor with shape (N, num_total_anchors)

• bbox_targets (Tensor) – BBox regression targets of each anchor weight shape (N, num_total_anchors, 4).

• bbox_weights (Tensor) – BBox regression loss weights of each anchor with shape (N, num_total_anchors, 4).

• avg_factor (int) – Average factor that is used to average the loss.

Returns

loss components.

Return type

tuple

class mmdet.models.dense_heads.AutoAssignHead(*args, force_topk: bool = False, topk: int = 9, pos_loss_weight: float = 0.25, neg_loss_weight: float = 0.75, center_loss_weight: float = 0.75, **kwargs)

AutoAssignHead head used in AutoAssign.

More details can be found in the paper .

Parameters

• force_topk (bool) – Used in center prior initialization to handle extremely small gt. Default is False.

• topk (int) – The number of points used to calculate the center prior when no point falls in gt_bbox. Only work when force_topk if True. Defaults to 9.

• pos_loss_weight (float) – The loss weight of positive loss and with default value 0.25.

• neg_loss_weight (float) – The loss weight of negative loss and with default value 0.75.

• center_loss_weight (float) – The loss weight of center prior loss and with default value 0.75.

forward_single(x: Tensor, scale: Scale, stride: int) → Tuple[Tensor, Tensor, Tensor]

Forward features of a single scale level.

Parameters

• x (Tensor) – FPN feature maps of the specified stride.

• scale (mmcv.cnn.Scale) – Learnable scale module to resize the bbox prediction.

• stride (int) – The corresponding stride for feature maps, only used to normalize the bbox prediction when self.norm_on_bbox is True.

Returns

scores for each class, bbox predictions and centerness predictions of input feature maps.

Return type

tuple[Tensor, Tensor, Tensor]

get_neg_loss_single(cls_score: Tensor, objectness: Tensor, gt_instances: InstanceData, ious: Tensor, inside_gt_bbox_mask: Tensor) → Tuple[Tensor]

Calculate the negative loss of all points in feature map.

Parameters

• cls_score (Tensor) – All category scores for each point on the feature map. The shape is (num_points, num_class).

• objectness (Tensor) – Foreground probability of all points and is shape of (num_points, 1).

• gt_instances (InstanceData) – Ground truth of instance annotations. It should includes bboxes and labels attributes.

• ious (Tensor) – Float tensor with shape of (num_points, num_gt). Each value represent the iou of pred_bbox and gt_bboxes.

• inside_gt_bbox_mask (Tensor) – Tensor of bool type, with shape of (num_points, num_gt), each value is used to mark whether this point falls within a certain gt.

Returns

• neg_loss (Tensor): The negative loss of all points in the feature map.

Return type

tuple[Tensor]

get_pos_loss_single(cls_score: Tensor, objectness: Tensor, reg_loss: Tensor, gt_instances: InstanceData, center_prior_weights: Tensor) → Tuple[Tensor]

Calculate the positive loss of all points in gt_bboxes.

Parameters

• cls_score (Tensor) – All category scores for each point on the feature map. The shape is (num_points, num_class).

• objectness (Tensor) – Foreground probability of all points, has shape (num_points, 1).

• reg_loss (Tensor) – The regression loss of each gt_bbox and each prediction box, has shape of (num_points, num_gt).

• gt_instances (InstanceData) – Ground truth of instance annotations. It should includes bboxes and labels attributes.

• center_prior_weights (Tensor) – Float tensor with shape of (num_points, num_gt). Each value represents the center weighting coefficient.

Returns

• pos_loss (Tensor): The positive loss of all points in the gt_bboxes.

Return type

tuple[Tensor]

get_targets(points: List[Tensor], batch_gt_instances: List[InstanceData]) → Tuple[List[Tensor], List[Tensor]]

Compute regression targets and each point inside or outside gt_bbox in multiple images.

Parameters

• points (list[Tensor]) – Points of all fpn level, each has shape (num_points, 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

Returns

• inside_gt_bbox_mask_list (list[Tensor]): Each Tensor is with bool type and shape of (num_points, num_gt), each value is used to mark whether this point falls within a certain gt.

• concat_lvl_bbox_targets (list[Tensor]): BBox targets of each level. Each tensor has shape (num_points, num_gt, 4).

Return type

tuple(list[Tensor], list[Tensor])

init_weights() → None

Initialize weights of the head.

In particular, we have special initialization for classified conv’s and regression conv’s bias

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], objectnesses: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level, each is a 4D-tensor, the channel number is num_points * num_classes.

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

• objectnesses (list[Tensor]) – objectness for each scale level, each is a 4D-tensor, the channel number is num_points * 1.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.BoxInstBboxHead(*args, **kwargs)

BoxInst box head used in https://arxiv.org/abs/2012.02310.

class mmdet.models.dense_heads.BoxInstMaskHead(*arg, pairwise_size: int = 3, pairwise_dilation: int = 2, warmup_iters: int = 10000, **kwargs)

BoxInst mask head used in https://arxiv.org/abs/2012.02310.

This head outputs the mask for BoxInst.

Parameters

• pairwise_size (dict) – The size of neighborhood for each pixel. Defaults to 3.

• pairwise_dilation (int) – The dilation of neighborhood for each pixel. Defaults to 2.

• warmup_iters (int) – Warmup iterations for pair-wise loss. Defaults to 10000.

get_pairwise_affinity(mask_logits: Tensor) → Tensor

Compute the pairwise affinity for each pixel.

loss_by_feat(mask_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], positive_infos: List[InstanceData], **kwargs) → dict

Calculate the loss based on the features extracted by the mask head.

Parameters

• mask_preds (list[Tensor]) – List of predicted masks, each has shape (num_classes, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, masks, and labels attributes.

• batch_img_metas (list[dict]) – Meta information of multiple images.

• positive_infos (List[:obj:InstanceData]) – Information of positive samples of each image that are assigned in detection head.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.CLRHead(num_points: int = 72, prior_feat_channels: int = 64, fc_hidden_dim: int = 64, num_priors: int = 192, num_fc: int = 2, refine_layers: int = 3, sample_points: int = 36, img_w: int = 800, img_h: int = 320, num_classes: int = 5, cut_height: int = 0, cls_loss: Optional[dict] = None, cls_loss_weight: float = 2.0, xyt_loss_weight: float = 0.5, iou_loss_weight: float = 2.0, seg_loss_weight: float = 1.0, num_classes_seg: int = 5, train_cfg=None, test_cfg=None)

forward(inputs, batch_data_samples, training: bool = False, **kwargs)

Take pyramid features as input to perform Cross Layer Refinement and finally output the prediction lanes.

Each feature is a 4D tensor. :param x: input features (list[Tensor])

Returns

each layer’s prediction result seg: segmentation result for auxiliary loss

Return type

prediction_list

get_lanes(output, org_widths, as_lanes=True, conf_threshold=0.05, nms_threshold=0.5, top_k=5)

Convert model output to lanes.

pool_prior_features(batch_features, num_priors, prior_xs)

Pool prior feature from feature map.

Parameters

batch_features (Tensor) – Input feature maps, shape: (B, C, H, W)

predict(x, batched_data_samples, *args, **kwargs)

Perform inference on the input image.

Parameters

x – input image

Returns

list of lanes

Return type

lanes

predictions_to_pred(predictions, scores, org_width)

Convert predictions to internal Lane structure for evaluation.

class mmdet.models.dense_heads.CascadeRPNHead(num_classes: int, num_stages: int, stages: List[Union[ConfigDict, dict]], train_cfg: List[Union[ConfigDict, dict]], test_cfg: Union[ConfigDict, dict], init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

The CascadeRPNHead will predict more accurate region proposals, which is required for two-stage detectors (such as Fast/Faster R-CNN). CascadeRPN consists of a sequence of RPNStage to progressively improve the accuracy of the detected proposals.

More details can be found in https://arxiv.org/abs/1909.06720.

Parameters

• num_stages (int) – number of CascadeRPN stages.

• stages (list[ConfigDict or dict]) – list of configs to build the stages.

• train_cfg (list[ConfigDict or dict]) – list of configs at training time each stage.

• test_cfg (ConfigDict or dict) – config at testing time.

• init_cfg (ConfigDict or list[ConfigDict] or dict or list[dict]) – Initialization config dict.

loss(x: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

loss_and_predict(x: Tuple[Tensor], batch_data_samples: List[DetDataSample], proposal_cfg: Optional[ConfigDict] = None) → Tuple[dict, List[InstanceData]]

Perform forward propagation of the head, then calculate loss and predictions from the features and data samples.

Parameters

• x (tuple[Tensor]) – Features from FPN.

• batch_data_samples (list[DetDataSample]) – Each item contains the meta information of each image and corresponding annotations.

• proposal_cfg (ConfigDict, optional) – Test / postprocessing configuration, if None, test_cfg would be used. Defaults to None.

Returns

the return value is a tuple contains:

• losses: (dict[str, Tensor]): A dictionary of loss components.

• predictions (list[InstanceData]): Detection results of each image after the post process.

Return type

tuple

loss_by_feat()

loss_by_feat() is implemented in StageCascadeRPNHead.

predict(x: Tuple[Tensor], batch_data_samples: List[DetDataSample], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the detection head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Multi-level features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool, optional) – Whether to rescale the results. Defaults to False.

Returns

InstanceData]: Detection results of each image after the post process.

Return type

list[obj

predict_by_feat()

predict_by_feat() is implemented in StageCascadeRPNHead.

class mmdet.models.dense_heads.CenterNetHead(in_channels: int, feat_channels: int, num_classes: int, loss_center_heatmap: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'GaussianFocalLoss'}, loss_wh: Union[ConfigDict, dict] = {'loss_weight': 0.1, 'type': 'L1Loss'}, loss_offset: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'L1Loss'}, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Objects as Points Head. CenterHead use center_point to indicate object’s position. Paper link <https://arxiv.org/abs/1904.07850>

Parameters

• in_channels (int) – Number of channel in the input feature map.

• feat_channels (int) – Number of channel in the intermediate feature map.

• num_classes (int) – Number of categories excluding the background category.

• loss_center_heatmap (ConfigDict or dict) – Config of center heatmap loss. Defaults to dict(type=’GaussianFocalLoss’, loss_weight=1.0)

• loss_wh (ConfigDict or dict) – Config of wh loss. Defaults to dict(type=’L1Loss’, loss_weight=0.1).

• loss_offset (ConfigDict or dict) – Config of offset loss. Defaults to dict(type=’L1Loss’, loss_weight=1.0).

• train_cfg (ConfigDict or dict, optional) – Training config. Useless in CenterNet, but we keep this variable for SingleStageDetector.

• test_cfg (ConfigDict or dict, optional) – Testing config of CenterNet.

:param init_cfg (ConfigDict or dict or list[dict] or: list[ConfigDict], optional): Initialization

config dict.

forward(x: Tuple[Tensor, ...]) → Tuple[List[Tensor]]

Forward features. Notice CenterNet head does not use FPN.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

center predict heatmaps for

all levels, the channels number is num_classes.

wh_preds (list[Tensor]): wh predicts for all levels, the channels

number is 2.

offset_preds (list[Tensor]): offset predicts for all levels, the

channels number is 2.

Return type

center_heatmap_preds (list[Tensor])

forward_single(x: Tensor) → Tuple[Tensor, ...]

Forward feature of a single level.

Parameters

x (Tensor) – Feature of a single level.

Returns

center predict heatmaps, the

channels number is num_classes.

wh_pred (Tensor): wh predicts, the channels number is 2. offset_pred (Tensor): offset predicts, the channels number is 2.

Return type

center_heatmap_pred (Tensor)

get_targets(gt_bboxes: List[Tensor], gt_labels: List[Tensor], feat_shape: tuple, img_shape: tuple) → Tuple[dict, int]

Compute regression and classification targets in multiple images.

Parameters

• gt_bboxes (list[Tensor]) – Ground truth bboxes for each image with shape (num_gts, 4) in [tl_x, tl_y, br_x, br_y] format.

• gt_labels (list[Tensor]) – class indices corresponding to each box.

• feat_shape (tuple) – feature map shape with value [B, _, H, W]

• img_shape (tuple) – image shape.

Returns

The float value is mean avg_factor, the dict has components below:

• center_heatmap_target (Tensor): targets of center heatmap, shape (B, num_classes, H, W).

• wh_target (Tensor): targets of wh predict, shape (B, 2, H, W).

• offset_target (Tensor): targets of offset predict, shape (B, 2, H, W).

• wh_offset_target_weight (Tensor): weights of wh and offset predict, shape (B, 2, H, W).

Return type

tuple[dict, float]

init_weights() → None

Initialize weights of the head.

loss_by_feat(center_heatmap_preds: List[Tensor], wh_preds: List[Tensor], offset_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Compute losses of the head.

Parameters

• center_heatmap_preds (list[Tensor]) – center predict heatmaps for all levels with shape (B, num_classes, H, W).

• wh_preds (list[Tensor]) – wh predicts for all levels with shape (B, 2, H, W).

• offset_preds (list[Tensor]) – offset predicts for all levels with shape (B, 2, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

which has components below:

• loss_center_heatmap (Tensor): loss of center heatmap.

• loss_wh (Tensor): loss of hw heatmap

• loss_offset (Tensor): loss of offset heatmap.

Return type

dict[str, Tensor]

predict_by_feat(center_heatmap_preds: List[Tensor], wh_preds: List[Tensor], offset_preds: List[Tensor], batch_img_metas: Optional[List[dict]] = None, rescale: bool = True, with_nms: bool = False) → List[InstanceData]

Transform network output for a batch into bbox predictions.

Parameters

• center_heatmap_preds (list[Tensor]) – Center predict heatmaps for all levels with shape (B, num_classes, H, W).

• wh_preds (list[Tensor]) – WH predicts for all levels with shape (B, 2, H, W).

• offset_preds (list[Tensor]) – Offset predicts for all levels with shape (B, 2, H, W).

• batch_img_metas (list[dict], optional) – Batch image meta info. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to True.

• with_nms (bool) – If True, do nms before return boxes. Defaults to False.

Returns

Instance segmentation results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.dense_heads.CenterNetUpdateHead(num_classes: int, in_channels: int, regress_ranges: Sequence[Tuple[int, int]] = ((0, 80), (64, 160), (128, 320), (256, 640), (512, 1000000000)), hm_min_radius: int = 4, hm_min_overlap: float = 0.8, more_pos_thresh: float = 0.2, more_pos_topk: int = 9, soft_weight_on_reg: bool = False, loss_cls: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'neg_weight': 0.75, 'pos_weight': 0.25, 'type': 'GaussianFocalLoss'}, loss_bbox: Union[ConfigDict, dict] = {'loss_weight': 2.0, 'type': 'GIoULoss'}, norm_cfg: Optional[Union[ConfigDict, dict]] = {'num_groups': 32, 'requires_grad': True, 'type': 'GN'}, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, **kwargs)

CenterNetUpdateHead is an improved version of CenterNet in CenterNet2. Paper link https://arxiv.org/abs/2103.07461.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channel in the input feature map.

• regress_ranges (Sequence[Tuple[int, int]]) – Regress range of multiple level points.

• hm_min_radius (int) – Heatmap target minimum radius of cls branch. Defaults to 4.

• hm_min_overlap (float) – Heatmap target minimum overlap of cls branch. Defaults to 0.8.

• more_pos_thresh (float) – The filtering threshold when the cls branch adds more positive samples. Defaults to 0.2.

• more_pos_topk (int) – The maximum number of additional positive samples added to each gt. Defaults to 9.

• soft_weight_on_reg (bool) – Whether to use the soft target of the cls branch as the soft weight of the bbox branch. Defaults to False.

• loss_cls (ConfigDict or dict) – Config of cls loss. Defaults to dict(type=’GaussianFocalLoss’, loss_weight=1.0)

• loss_bbox (ConfigDict or dict) – Config of bbox loss. Defaults to dict(type=’GIoULoss’, loss_weight=2.0).

• norm_cfg (ConfigDict or dict, optional) – dictionary to construct and config norm layer. Defaults to norm_cfg=dict(type='GN', num_groups=32, requires_grad=True).

• train_cfg (ConfigDict or dict, optional) – Training config. Unused in CenterNet. Reserved for compatibility with SingleStageDetector.

• test_cfg (ConfigDict or dict, optional) – Testing config of CenterNet.

add_cls_pos_inds(flatten_points: Tensor, flatten_bbox_preds: Tensor, featmap_sizes: Tensor, batch_gt_instances: List[InstanceData]) → Tuple[Optional[Tensor], Optional[Tensor]]

Provide additional adaptive positive samples to the classification branch.

Parameters

• flatten_points (Tensor) – The point after flatten, including batch image and all levels. The shape is (N, 2).

• flatten_bbox_preds (Tensor) – The bbox predicts after flatten, including batch image and all levels. The shape is (N, 4).

• featmap_sizes (Tensor) – Feature map size of all layers. The shape is (5, 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

Returns

• pos_inds (Tensor): Adaptively selected positive sample index.

• cls_labels (Tensor): Corresponding positive class label.

Return type

tuple

forward(x: Tuple[Tensor]) → Tuple[List[Tensor], List[Tensor]]

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of each level outputs.

• cls_scores (list[Tensor]): Box scores for each scale level, each is a 4D-tensor, the channel number is num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is 4.

Return type

tuple

forward_single(x: Tensor, scale: Scale, stride: int) → Tuple[Tensor, Tensor]

Forward features of a single scale level.

Parameters

• x (Tensor) – FPN feature maps of the specified stride.

• scale (mmcv.cnn.Scale) – Learnable scale module to resize the bbox prediction.

• stride (int) – The corresponding stride for feature maps.

Returns

scores for each class, bbox predictions of input feature maps.

Return type

tuple

get_targets(points: List[Tensor], batch_gt_instances: List[InstanceData]) → Tuple[Tensor, Tensor]

Compute classification and bbox targets for points in multiple images.

Parameters

• points (list[Tensor]) – Points of each fpn level, each has shape (num_points, 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

Returns

Targets of each level.

• concat_lvl_labels (Tensor): Labels of all level and batch.

• concat_lvl_bbox_targets (Tensor): BBox targets of all level and batch.

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level, each is a 4D-tensor, the channel number is num_classes.

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is 4.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.CentripetalHead(*args, centripetal_shift_channels: int = 2, guiding_shift_channels: int = 2, feat_adaption_conv_kernel: int = 3, loss_guiding_shift: Union[ConfigDict, dict] = {'beta': 1.0, 'loss_weight': 0.05, 'type': 'SmoothL1Loss'}, loss_centripetal_shift: Union[ConfigDict, dict] = {'beta': 1.0, 'loss_weight': 1, 'type': 'SmoothL1Loss'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, **kwargs)

Head of CentripetalNet: Pursuing High-quality Keypoint Pairs for Object Detection.

CentripetalHead inherits from CornerHead. It removes the embedding branch and adds guiding shift and centripetal shift branches. More details can be found in the paper .

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• num_feat_levels (int) – Levels of feature from the previous module. 2 for HourglassNet-104 and 1 for HourglassNet-52. HourglassNet-104 outputs the final feature and intermediate supervision feature and HourglassNet-52 only outputs the final feature. Defaults to 2.

• corner_emb_channels (int) – Channel of embedding vector. Defaults to 1.

• train_cfg (ConfigDict or dict, optional) – Training config. Useless in CornerHead, but we keep this variable for SingleStageDetector.

• test_cfg (ConfigDict or dict, optional) – Testing config of CornerHead.

• loss_heatmap (ConfigDict or dict) – Config of corner heatmap loss. Defaults to GaussianFocalLoss.

• loss_embedding (ConfigDict or dict) – Config of corner embedding loss. Defaults to AssociativeEmbeddingLoss.

• loss_offset (ConfigDict or dict) – Config of corner offset loss. Defaults to SmoothL1Loss.

• loss_guiding_shift (ConfigDict or dict) – Config of guiding shift loss. Defaults to SmoothL1Loss.

• loss_centripetal_shift – Config of centripetal shift loss. Defaults to SmoothL1Loss.

forward_single(x: Tensor, lvl_ind: int) → List[Tensor]

Forward feature of a single level.

Parameters

• x (Tensor) – Feature of a single level.

• lvl_ind (int) – Level index of current feature.

Returns

A tuple of CentripetalHead’s output for current feature level. Containing the following Tensors:

• tl_heat (Tensor): Predicted top-left corner heatmap.

• br_heat (Tensor): Predicted bottom-right corner heatmap.

• tl_off (Tensor): Predicted top-left offset heatmap.

• br_off (Tensor): Predicted bottom-right offset heatmap.

• tl_guiding_shift (Tensor): Predicted top-left guiding shift heatmap.

• br_guiding_shift (Tensor): Predicted bottom-right guiding shift heatmap.

• tl_centripetal_shift (Tensor): Predicted top-left centripetal shift heatmap.

• br_centripetal_shift (Tensor): Predicted bottom-right centripetal shift heatmap.

Return type

tuple[Tensor]

init_weights() → None

Initialize the weights.

loss_by_feat(tl_heats: List[Tensor], br_heats: List[Tensor], tl_offs: List[Tensor], br_offs: List[Tensor], tl_guiding_shifts: List[Tensor], br_guiding_shifts: List[Tensor], tl_centripetal_shifts: List[Tensor], br_centripetal_shifts: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• tl_heats (list[Tensor]) – Top-left corner heatmaps for each level with shape (N, num_classes, H, W).

• br_heats (list[Tensor]) – Bottom-right corner heatmaps for each level with shape (N, num_classes, H, W).

• tl_offs (list[Tensor]) – Top-left corner offsets for each level with shape (N, corner_offset_channels, H, W).

• br_offs (list[Tensor]) – Bottom-right corner offsets for each level with shape (N, corner_offset_channels, H, W).

• tl_guiding_shifts (list[Tensor]) – Top-left guiding shifts for each level with shape (N, guiding_shift_channels, H, W).

• br_guiding_shifts (list[Tensor]) – Bottom-right guiding shifts for each level with shape (N, guiding_shift_channels, H, W).

• tl_centripetal_shifts (list[Tensor]) – Top-left centripetal shifts for each level with shape (N, centripetal_shift_channels, H, W).

• br_centripetal_shifts (list[Tensor]) – Bottom-right centripetal shifts for each level with shape (N, centripetal_shift_channels, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Specify which bounding boxes can be ignored when computing the loss.

Returns

A dictionary of loss components. Containing the following losses:

• det_loss (list[Tensor]): Corner keypoint losses of all feature levels.

• off_loss (list[Tensor]): Corner offset losses of all feature levels.

• guiding_loss (list[Tensor]): Guiding shift losses of all feature levels.

• centripetal_loss (list[Tensor]): Centripetal shift losses of all feature levels.

Return type

dict[str, Tensor]

loss_by_feat_single(tl_hmp: Tensor, br_hmp: Tensor, tl_off: Tensor, br_off: Tensor, tl_guiding_shift: Tensor, br_guiding_shift: Tensor, tl_centripetal_shift: Tensor, br_centripetal_shift: Tensor, targets: dict) → Tuple[Tensor, ...]

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• tl_hmp (Tensor) – Top-left corner heatmap for current level with shape (N, num_classes, H, W).

• br_hmp (Tensor) – Bottom-right corner heatmap for current level with shape (N, num_classes, H, W).

• tl_off (Tensor) – Top-left corner offset for current level with shape (N, corner_offset_channels, H, W).

• br_off (Tensor) – Bottom-right corner offset for current level with shape (N, corner_offset_channels, H, W).

• tl_guiding_shift (Tensor) – Top-left guiding shift for current level with shape (N, guiding_shift_channels, H, W).

• br_guiding_shift (Tensor) – Bottom-right guiding shift for current level with shape (N, guiding_shift_channels, H, W).

• tl_centripetal_shift (Tensor) – Top-left centripetal shift for current level with shape (N, centripetal_shift_channels, H, W).

• br_centripetal_shift (Tensor) – Bottom-right centripetal shift for current level with shape (N, centripetal_shift_channels, H, W).

• targets (dict) – Corner target generated by get_targets.

Returns

Losses of the head’s different branches containing the following losses:

• det_loss (Tensor): Corner keypoint loss.

• off_loss (Tensor): Corner offset loss.

• guiding_loss (Tensor): Guiding shift loss.

• centripetal_loss (Tensor): Centripetal shift loss.

Return type

tuple[torch.Tensor]

predict_by_feat(tl_heats: List[Tensor], br_heats: List[Tensor], tl_offs: List[Tensor], br_offs: List[Tensor], tl_guiding_shifts: List[Tensor], br_guiding_shifts: List[Tensor], tl_centripetal_shifts: List[Tensor], br_centripetal_shifts: List[Tensor], batch_img_metas: Optional[List[dict]] = None, rescale: bool = False, with_nms: bool = True) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Parameters

• tl_heats (list[Tensor]) – Top-left corner heatmaps for each level with shape (N, num_classes, H, W).

• br_heats (list[Tensor]) – Bottom-right corner heatmaps for each level with shape (N, num_classes, H, W).

• tl_offs (list[Tensor]) – Top-left corner offsets for each level with shape (N, corner_offset_channels, H, W).

• br_offs (list[Tensor]) – Bottom-right corner offsets for each level with shape (N, corner_offset_channels, H, W).

• tl_guiding_shifts (list[Tensor]) – Top-left guiding shifts for each level with shape (N, guiding_shift_channels, H, W). Useless in this function, we keep this arg because it’s the raw output from CentripetalHead.

• br_guiding_shifts (list[Tensor]) – Bottom-right guiding shifts for each level with shape (N, guiding_shift_channels, H, W). Useless in this function, we keep this arg because it’s the raw output from CentripetalHead.

• tl_centripetal_shifts (list[Tensor]) – Top-left centripetal shifts for each level with shape (N, centripetal_shift_channels, H, W).

• br_centripetal_shifts (list[Tensor]) – Bottom-right centripetal shifts for each level with shape (N, centripetal_shift_channels, H, W).

• batch_img_metas (list[dict], optional) – Batch image meta info. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

• with_nms (bool) – If True, do nms before return boxes. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.dense_heads.CondInstBboxHead(*args, num_params: int = 169, **kwargs)

CondInst box head used in https://arxiv.org/abs/1904.02689.

Note that CondInst Bbox Head is a extension of FCOS head. Two differences are described as follows:

1. CondInst box head predicts a set of params for each instance.

2. CondInst box head return the pos_gt_inds and pos_inds.

Parameters

num_params (int) – Number of params for instance segmentation.

forward_single(x: Tensor, scale: Scale, stride: int) → Tuple[Tensor, Tensor, Tensor, Tensor]

Forward features of a single scale level.

Parameters

• x (Tensor) – FPN feature maps of the specified stride.

• scale (mmcv.cnn.Scale) – Learnable scale module to resize the bbox prediction.

• stride (int) – The corresponding stride for feature maps, only used to normalize the bbox prediction when self.norm_on_bbox is True.

Returns

scores for each class, bbox predictions, centerness predictions and param predictions of input feature maps.

Return type

tuple

get_positive_infos() → List[InstanceData]

Get positive information from sampling results.

Returns

Positive information of each image, usually including positive bboxes, positive labels, positive priors, etc.

Return type

list[InstanceData]

get_targets(points: List[Tensor], batch_gt_instances: List[InstanceData]) → Tuple[List[Tensor], List[Tensor], List[Tensor], List[Tensor]]

Compute regression, classification and centerness targets for points in multiple images.

Parameters

• points (list[Tensor]) – Points of each fpn level, each has shape (num_points, 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

Returns

Targets of each level.

• concat_lvl_labels (list[Tensor]): Labels of each level.

• concat_lvl_bbox_targets (list[Tensor]): BBox targets of each level.

• pos_inds_list (list[Tensor]): pos_inds of each image.

• pos_gt_inds_list (List[Tensor]): pos_gt_inds of each image.

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], centernesses: List[Tensor], param_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level, each is a 4D-tensor, the channel number is num_points * num_classes.

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

• centernesses (list[Tensor]) – centerness for each scale level, each is a 4D-tensor, the channel number is num_points * 1.

• param_preds (List[Tensor]) – param_pred for each scale level, each is a 4D-tensor, the channel number is num_params.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

predict_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], score_factors: Optional[List[Tensor]] = None, param_preds: Optional[List[Tensor]] = None, batch_img_metas: Optional[List[dict]] = None, cfg: Optional[ConfigDict] = None, rescale: bool = False, with_nms: bool = True) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Note: When score_factors is not None, the cls_scores are usually multiplied by it then obtain the real score used in NMS, such as CenterNess in FCOS, IoU branch in ATSS.

Parameters

• cls_scores (list[Tensor]) – Classification scores for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * 4, H, W).

• score_factors (list[Tensor], optional) – Score factor for all scale level, each is a 4D-tensor, has shape (batch_size, num_priors * 1, H, W). Defaults to None.

• param_preds (list[Tensor], optional) – Params for all scale level, each is a 4D-tensor, has shape (batch_size, num_priors * num_params, H, W)

• batch_img_metas (list[dict], Optional) – Batch image meta info. Defaults to None.

• cfg (ConfigDict, optional) – Test / postprocessing configuration, if None, test_cfg would be used. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

• with_nms (bool) – If True, do nms before return boxes. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.dense_heads.CondInstMaskHead(mask_feature_head: Union[ConfigDict, dict], num_layers: int = 3, feat_channels: int = 8, mask_out_stride: int = 4, size_of_interest: int = 8, max_masks_to_train: int = -1, topk_masks_per_img: int = -1, loss_mask: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None)

CondInst mask head used in https://arxiv.org/abs/1904.02689.

This head outputs the mask for CondInst.

Parameters

• mask_feature_head (dict) – Config of CondInstMaskFeatHead.

• num_layers (int) – Number of dynamic conv layers.

• feat_channels (int) – Number of channels in the dynamic conv.

• mask_out_stride (int) – The stride of the mask feat.

• size_of_interest (int) – The size of the region used in rel coord.

• max_masks_to_train (int) – Maximum number of masks to train for each image.

• loss_segm (ConfigDict or dict, optional) – Config of segmentation loss.

• train_cfg (ConfigDict or dict, optional) – Training config of head.

• test_cfg (ConfigDict or dict, optional) – Testing config of head.

dynamic_conv_forward(features: Tensor, weights: List[Tensor], biases: List[Tensor], num_insts: int) → Tensor

Dynamic forward, each layer follow a relu.

forward(x: tuple, positive_infos: List[InstanceData]) → tuple

Forward feature from the upstream network to get prototypes and linearly combine the prototypes, using masks coefficients, into instance masks. Finally, crop the instance masks with given bboxes.

Parameters

• x (Tuple[Tensor]) – Feature from the upstream network, which is a 4D-tensor.

• positive_infos (List[:obj:InstanceData]) – Positive information that calculate from detect head.

Returns

Predicted instance segmentation masks

Return type

tuple

forward_single(mask_feat: Tensor, positive_info: InstanceData) → Tensor

Forward features of a each image.

loss_by_feat(mask_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], positive_infos: List[InstanceData], **kwargs) → dict

Calculate the loss based on the features extracted by the mask head.

Parameters

• mask_preds (list[Tensor]) – List of predicted masks, each has shape (num_classes, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, masks, and labels attributes.

• batch_img_metas (list[dict]) – Meta information of multiple images.

• positive_infos (List[:obj:InstanceData]) – Information of positive samples of each image that are assigned in detection head.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

parse_dynamic_params(params: Tensor) → Tuple[List[Tensor], List[Tensor]]

Parse the dynamic params for dynamic conv.

predict_by_feat(mask_preds: List[Tensor], results_list: List[InstanceData], batch_img_metas: List[dict], rescale: bool = True, **kwargs) → List[InstanceData]

Transform a batch of output features extracted from the head into mask results.

Parameters

• mask_preds (list[Tensor]) – Predicted prototypes with shape (num_classes, H, W).

• results_list (List[:obj:InstanceData]) – BBoxHead results.

• batch_img_metas (list[dict]) – Meta information of all images.

• rescale (bool, optional) – Whether to rescale the results. Defaults to False.

Returns

Processed results of multiple images.Each InstanceData usually contains following keys.

• scores (Tensor): Classification scores, has shape (num_instance,).

• labels (Tensor): Has shape (num_instances,).

• masks (Tensor): Processed mask results, has shape (num_instances, h, w).

Return type

list[InstanceData]

class mmdet.models.dense_heads.ConditionalDETRHead(num_classes: int, embed_dims: int = 256, num_reg_fcs: int = 2, sync_cls_avg_factor: bool = False, loss_cls: Union[ConfigDict, dict] = {'bg_cls_weight': 0.1, 'class_weight': 1.0, 'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': False}, loss_bbox: Union[ConfigDict, dict] = {'loss_weight': 5.0, 'type': 'L1Loss'}, loss_iou: Union[ConfigDict, dict] = {'loss_weight': 2.0, 'type': 'GIoULoss'}, train_cfg: Union[ConfigDict, dict] = {'assigner': {'match_costs': [{'type': 'ClassificationCost', 'weight': 1.0}, {'type': 'BBoxL1Cost', 'weight': 5.0, 'box_format': 'xywh'}, {'type': 'IoUCost', 'iou_mode': 'giou', 'weight': 2.0}], 'type': 'HungarianAssigner'}}, test_cfg: Union[ConfigDict, dict] = {'max_per_img': 100}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Head of Conditional DETR. Conditional DETR: Conditional DETR for Fast Training Convergence. More details can be found in the `paper.

<https://arxiv.org/abs/2108.06152>`_ .

forward(hidden_states: Tensor, references: Tensor) → Tuple[Tensor, Tensor]

“Forward function.

Parameters

• hidden_states (Tensor) – Features from transformer decoder. If return_intermediate_dec is True output has shape (num_decoder_layers, bs, num_queries, dim), else has shape (1, bs, num_queries, dim) which only contains the last layer outputs.

• references (Tensor) – References from transformer decoder, has shape (bs, num_queries, 2).

Returns

results of head containing the following tensor.

• layers_cls_scores (Tensor): Outputs from the classification head, shape (num_decoder_layers, bs, num_queries, cls_out_channels). Note cls_out_channels should include background.

• layers_bbox_preds (Tensor): Sigmoid outputs from the regression head with normalized coordinate format (cx, cy, w, h), has shape (num_decoder_layers, bs, num_queries, 4).

Return type

tuple[Tensor]

init_weights()

Initialize weights of the transformer head.

loss(hidden_states: Tensor, references: Tensor, batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection head on the features of the upstream network.

Parameters

• hidden_states (Tensor) – Features from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, dim).

• references (Tensor) – References from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, 2).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

loss_and_predict(hidden_states: Tensor, references: Tensor, batch_data_samples: List[DetDataSample]) → Tuple[dict, List[InstanceData]]

Perform forward propagation of the head, then calculate loss and predictions from the features and data samples. Over-write because img_metas are needed as inputs for bbox_head.

Parameters

• hidden_states (Tensor) – Features from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, dim).

• references (Tensor) – References from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, 2).

• batch_data_samples (list[DetDataSample]) – Each item contains the meta information of each image and corresponding annotations.

Returns

The return value is a tuple contains:

• losses: (dict[str, Tensor]): A dictionary of loss components.

• predictions (list[InstanceData]): Detection results of each image after the post process.

Return type

tuple

predict(hidden_states: Tensor, references: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[InstanceData]

Perform forward propagation of the detection head and predict detection results on the features of the upstream network. Over-write because img_metas are needed as inputs for bbox_head.

Parameters

• hidden_states (Tensor) – Features from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, dim).

• references (Tensor) – References from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, 2).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool, optional) – Whether to rescale the results. Defaults to True.

Returns

InstanceData]: Detection results of each image after the post process.

Return type

list[obj

class mmdet.models.dense_heads.CornerHead(num_classes: int, in_channels: int, num_feat_levels: int = 2, corner_emb_channels: int = 1, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, loss_heatmap: Union[ConfigDict, dict] = {'alpha': 2.0, 'gamma': 4.0, 'loss_weight': 1, 'type': 'GaussianFocalLoss'}, loss_embedding: Union[ConfigDict, dict] = {'pull_weight': 0.25, 'push_weight': 0.25, 'type': 'AssociativeEmbeddingLoss'}, loss_offset: Union[ConfigDict, dict] = {'beta': 1.0, 'loss_weight': 1, 'type': 'SmoothL1Loss'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Head of CornerNet: Detecting Objects as Paired Keypoints.

Code is modified from the official github repo .

More details can be found in the paper .

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• num_feat_levels (int) – Levels of feature from the previous module. 2 for HourglassNet-104 and 1 for HourglassNet-52. Because HourglassNet-104 outputs the final feature and intermediate supervision feature and HourglassNet-52 only outputs the final feature. Defaults to 2.

• corner_emb_channels (int) – Channel of embedding vector. Defaults to 1.

• train_cfg (ConfigDict or dict, optional) – Training config. Useless in CornerHead, but we keep this variable for SingleStageDetector.

• test_cfg (ConfigDict or dict, optional) – Testing config of CornerHead.

• loss_heatmap (ConfigDict or dict) – Config of corner heatmap loss. Defaults to GaussianFocalLoss.

• loss_embedding (ConfigDict or dict) – Config of corner embedding loss. Defaults to AssociativeEmbeddingLoss.

• loss_offset (ConfigDict or dict) – Config of corner offset loss. Defaults to SmoothL1Loss.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization.

forward(feats: Tuple[Tensor]) → tuple

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually a tuple of corner heatmaps, offset heatmaps and embedding heatmaps.

• tl_heats (list[Tensor]): Top-left corner heatmaps for all levels, each is a 4D-tensor, the channels number is num_classes.

• br_heats (list[Tensor]): Bottom-right corner heatmaps for all levels, each is a 4D-tensor, the channels number is num_classes.

• tl_embs (list[Tensor] | list[None]): Top-left embedding heatmaps for all levels, each is a 4D-tensor or None. If not None, the channels number is corner_emb_channels.

• br_embs (list[Tensor] | list[None]): Bottom-right embedding heatmaps for all levels, each is a 4D-tensor or None. If not None, the channels number is corner_emb_channels.

• tl_offs (list[Tensor]): Top-left offset heatmaps for all levels, each is a 4D-tensor. The channels number is corner_offset_channels.

• br_offs (list[Tensor]): Bottom-right offset heatmaps for all levels, each is a 4D-tensor. The channels number is corner_offset_channels.

Return type

tuple

forward_single(x: Tensor, lvl_ind: int, return_pool: bool = False) → List[Tensor]

Forward feature of a single level.

Parameters

• x (Tensor) – Feature of a single level.

• lvl_ind (int) – Level index of current feature.

• return_pool (bool) – Return corner pool feature or not. Defaults to False.

Returns

A tuple of CornerHead’s output for current feature level. Containing the following Tensors:

• tl_heat (Tensor): Predicted top-left corner heatmap.

• br_heat (Tensor): Predicted bottom-right corner heatmap.

• tl_emb (Tensor | None): Predicted top-left embedding heatmap. None for self.with_corner_emb == False.

• br_emb (Tensor | None): Predicted bottom-right embedding heatmap. None for self.with_corner_emb == False.

• tl_off (Tensor): Predicted top-left offset heatmap.

• br_off (Tensor): Predicted bottom-right offset heatmap.

• tl_pool (Tensor): Top-left corner pool feature. Not must have.

• br_pool (Tensor): Bottom-right corner pool feature. Not must have.

Return type

tuple[Tensor]

get_targets(gt_bboxes: List[Tensor], gt_labels: List[Tensor], feat_shape: Sequence[int], img_shape: Sequence[int], with_corner_emb: bool = False, with_guiding_shift: bool = False, with_centripetal_shift: bool = False) → dict

Generate corner targets.

Including corner heatmap, corner offset.

Optional: corner embedding, corner guiding shift, centripetal shift.

For CornerNet, we generate corner heatmap, corner offset and corner embedding from this function.

For CentripetalNet, we generate corner heatmap, corner offset, guiding shift and centripetal shift from this function.

Parameters

• gt_bboxes (list[Tensor]) – Ground truth bboxes of each image, each has shape (num_gt, 4).

• gt_labels (list[Tensor]) – Ground truth labels of each box, each has shape (num_gt, ).

• feat_shape (Sequence[int]) – Shape of output feature, [batch, channel, height, width].

• img_shape (Sequence[int]) – Shape of input image, [height, width, channel].

• with_corner_emb (bool) – Generate corner embedding target or not. Defaults to False.

• with_guiding_shift (bool) – Generate guiding shift target or not. Defaults to False.

• with_centripetal_shift (bool) – Generate centripetal shift target or not. Defaults to False.

Returns

Ground truth of corner heatmap, corner offset, corner embedding, guiding shift and centripetal shift. Containing the following keys:

• topleft_heatmap (Tensor): Ground truth top-left corner heatmap.

• bottomright_heatmap (Tensor): Ground truth bottom-right corner heatmap.

• topleft_offset (Tensor): Ground truth top-left corner offset.

• bottomright_offset (Tensor): Ground truth bottom-right corner offset.

• corner_embedding (list[list[list[int]]]): Ground truth corner embedding. Not must have.

• topleft_guiding_shift (Tensor): Ground truth top-left corner guiding shift. Not must have.

• bottomright_guiding_shift (Tensor): Ground truth bottom-right corner guiding shift. Not must have.

• topleft_centripetal_shift (Tensor): Ground truth top-left corner centripetal shift. Not must have.

• bottomright_centripetal_shift (Tensor): Ground truth bottom-right corner centripetal shift. Not must have.

Return type

dict

init_weights() → None

Initialize the weights.

loss_by_feat(tl_heats: List[Tensor], br_heats: List[Tensor], tl_embs: List[Tensor], br_embs: List[Tensor], tl_offs: List[Tensor], br_offs: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• tl_heats (list[Tensor]) – Top-left corner heatmaps for each level with shape (N, num_classes, H, W).

• br_heats (list[Tensor]) – Bottom-right corner heatmaps for each level with shape (N, num_classes, H, W).

• tl_embs (list[Tensor]) – Top-left corner embeddings for each level with shape (N, corner_emb_channels, H, W).

• br_embs (list[Tensor]) – Bottom-right corner embeddings for each level with shape (N, corner_emb_channels, H, W).

• tl_offs (list[Tensor]) – Top-left corner offsets for each level with shape (N, corner_offset_channels, H, W).

• br_offs (list[Tensor]) – Bottom-right corner offsets for each level with shape (N, corner_offset_channels, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Specify which bounding boxes can be ignored when computing the loss.

Returns

A dictionary of loss components. Containing the following losses:

• det_loss (list[Tensor]): Corner keypoint losses of all feature levels.

• pull_loss (list[Tensor]): Part one of AssociativeEmbedding losses of all feature levels.

• push_loss (list[Tensor]): Part two of AssociativeEmbedding losses of all feature levels.

• off_loss (list[Tensor]): Corner offset losses of all feature levels.

Return type

dict[str, Tensor]

loss_by_feat_single(tl_hmp: Tensor, br_hmp: Tensor, tl_emb: Optional[Tensor], br_emb: Optional[Tensor], tl_off: Tensor, br_off: Tensor, targets: dict) → Tuple[Tensor, ...]

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• tl_hmp (Tensor) – Top-left corner heatmap for current level with shape (N, num_classes, H, W).

• br_hmp (Tensor) – Bottom-right corner heatmap for current level with shape (N, num_classes, H, W).

• tl_emb (Tensor, optional) – Top-left corner embedding for current level with shape (N, corner_emb_channels, H, W).

• br_emb (Tensor, optional) – Bottom-right corner embedding for current level with shape (N, corner_emb_channels, H, W).

• tl_off (Tensor) – Top-left corner offset for current level with shape (N, corner_offset_channels, H, W).

• br_off (Tensor) – Bottom-right corner offset for current level with shape (N, corner_offset_channels, H, W).

• targets (dict) – Corner target generated by get_targets.

Returns

Losses of the head’s different branches containing the following losses:

• det_loss (Tensor): Corner keypoint loss.

• pull_loss (Tensor): Part one of AssociativeEmbedding loss.

• push_loss (Tensor): Part two of AssociativeEmbedding loss.

• off_loss (Tensor): Corner offset loss.

Return type

tuple[torch.Tensor]

predict_by_feat(tl_heats: List[Tensor], br_heats: List[Tensor], tl_embs: List[Tensor], br_embs: List[Tensor], tl_offs: List[Tensor], br_offs: List[Tensor], batch_img_metas: Optional[List[dict]] = None, rescale: bool = False, with_nms: bool = True) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Parameters

• tl_heats (list[Tensor]) – Top-left corner heatmaps for each level with shape (N, num_classes, H, W).

• br_heats (list[Tensor]) – Bottom-right corner heatmaps for each level with shape (N, num_classes, H, W).

• tl_embs (list[Tensor]) – Top-left corner embeddings for each level with shape (N, corner_emb_channels, H, W).

• br_embs (list[Tensor]) – Bottom-right corner embeddings for each level with shape (N, corner_emb_channels, H, W).

• tl_offs (list[Tensor]) – Top-left corner offsets for each level with shape (N, corner_offset_channels, H, W).

• br_offs (list[Tensor]) – Bottom-right corner offsets for each level with shape (N, corner_offset_channels, H, W).

• batch_img_metas (list[dict], optional) – Batch image meta info. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

• with_nms (bool) – If True, do nms before return boxes. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.dense_heads.DABDETRHead(num_classes: int, embed_dims: int = 256, num_reg_fcs: int = 2, sync_cls_avg_factor: bool = False, loss_cls: Union[ConfigDict, dict] = {'bg_cls_weight': 0.1, 'class_weight': 1.0, 'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': False}, loss_bbox: Union[ConfigDict, dict] = {'loss_weight': 5.0, 'type': 'L1Loss'}, loss_iou: Union[ConfigDict, dict] = {'loss_weight': 2.0, 'type': 'GIoULoss'}, train_cfg: Union[ConfigDict, dict] = {'assigner': {'match_costs': [{'type': 'ClassificationCost', 'weight': 1.0}, {'type': 'BBoxL1Cost', 'weight': 5.0, 'box_format': 'xywh'}, {'type': 'IoUCost', 'iou_mode': 'giou', 'weight': 2.0}], 'type': 'HungarianAssigner'}}, test_cfg: Union[ConfigDict, dict] = {'max_per_img': 100}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Head of DAB-DETR. DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR.

More details can be found in the paper .

forward(hidden_states: Tensor, references: Tensor) → Tuple[Tensor, Tensor]

“Forward function.

Parameters

• hidden_states (Tensor) – Features from transformer decoder. If return_intermediate_dec is True output has shape (num_decoder_layers, bs, num_queries, dim), else has shape (1, bs, num_queries, dim) which only contains the last layer outputs.

• references (Tensor) – References from transformer decoder. If return_intermediate_dec is True output has shape (num_decoder_layers, bs, num_queries, 2/4), else has shape (1, bs, num_queries, 2/4) which only contains the last layer reference.

Returns

results of head containing the following tensor.

• layers_cls_scores (Tensor): Outputs from the classification head, shape (num_decoder_layers, bs, num_queries, cls_out_channels). Note cls_out_channels should include background.

• layers_bbox_preds (Tensor): Sigmoid outputs from the regression head with normalized coordinate format (cx, cy, w, h), has shape (num_decoder_layers, bs, num_queries, 4).

Return type

tuple[Tensor]

init_weights() → None

Initialize weights.

predict(hidden_states: Tensor, references: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[InstanceData]

Perform forward propagation of the detection head and predict detection results on the features of the upstream network. Over-write because img_metas are needed as inputs for bbox_head.

Parameters

• hidden_states (Tensor) – Feature from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, dim).

• references (Tensor) – references from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, 2/4).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool, optional) – Whether to rescale the results. Defaults to True.

Returns

InstanceData]: Detection results of each image after the post process.

Return type

list[obj

class mmdet.models.dense_heads.DDODHead(num_classes: int, in_channels: int, stacked_convs: int = 4, conv_cfg: Optional[Union[ConfigDict, dict]] = None, use_dcn: bool = True, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 32, 'requires_grad': True, 'type': 'GN'}, loss_iou: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, **kwargs)

Detection Head of DDOD.

DDOD head decomposes conjunctions lying in most current one-stage detectors via label assignment disentanglement, spatial feature disentanglement, and pyramid supervision disentanglement.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• stacked_convs (int) – The number of stacked Conv. Defaults to 4.

• conv_cfg (ConfigDict or dict, optional) – Config dict for convolution layer. Defaults to None.

• use_dcn (bool) – Use dcn, Same as ATSS when False. Defaults to True.

• norm_cfg (ConfigDict or dict) – Normal config of ddod head. Defaults to dict(type=’GN’, num_groups=32, requires_grad=True).

• loss_iou (ConfigDict or dict) – Config of IoU loss. Defaults to dict(type=’CrossEntropyLoss’, use_sigmoid=True, loss_weight=1.0).

calc_reweight_factor(labels_list: List[Tensor]) → List[float]

Compute reweight_factor for regression and classification loss.

forward(x: Tuple[Tensor]) → Tuple[List[Tensor]]

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of classification scores, bbox predictions, and iou predictions.

• cls_scores (list[Tensor]): Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

• iou_preds (list[Tensor]): IoU scores for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 1.

Return type

tuple

forward_single(x: Tensor, scale: Scale) → Sequence[Tensor]

Forward feature of a single scale level.

Parameters

• x (Tensor) – Features of a single scale level.

• ( (scale) – obj: mmcv.cnn.Scale): Learnable scale module to resize the bbox prediction.

Returns

• cls_score (Tensor): Cls scores for a single scale level the channels number is num_base_priors * num_classes.

• bbox_pred (Tensor): Box energies / deltas for a single scale level, the channels number is num_base_priors * 4.

• iou_pred (Tensor): Iou for a single scale level, the channel number is (N, num_base_priors * 1, H, W).

Return type

tuple

get_cls_targets(anchor_list: List[Tensor], valid_flag_list: List[Tensor], num_level_anchors_list: List[int], cls_score_list: List[Tensor], bbox_pred_list: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs: bool = True) → tuple

Get cls targets for DDOD head.

This method is almost the same as AnchorHead.get_targets(). Besides returning the targets as the parent method does, it also returns the anchors as the first element of the returned tuple.

Parameters

• anchor_list (list[Tensor]) – anchors of each image.

• valid_flag_list (list[Tensor]) – Valid flags of each image.

• num_level_anchors_list (list[Tensor]) – Number of anchors of each scale level of all image.

• cls_score_list (list[Tensor]) – Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_pred_list (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• unmap_outputs (bool) – Whether to map outputs back to the original set of anchors.

Returns

A tuple of cls targets components.

Return type

tuple[Tensor]

get_num_level_anchors_inside(num_level_anchors: List[int], inside_flags: Tensor) → List[int]

Get the anchors of each scale level inside.

Parameters

• num_level_anchors (list[int]) – Number of anchors of each scale level.

• inside_flags (Tensor) – Multi level inside flags of the image, which are concatenated into a single tensor of shape (num_base_priors,).

Returns

Number of anchors of each scale level inside.

Return type

list[int]

get_reg_targets(anchor_list: List[Tensor], valid_flag_list: List[Tensor], num_level_anchors_list: List[int], cls_score_list: List[Tensor], bbox_pred_list: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs: bool = True) → tuple

Get reg targets for DDOD head.

This method is almost the same as AnchorHead.get_targets() when is_cls_assigner is False. Besides returning the targets as the parent method does, it also returns the anchors as the first element of the returned tuple.

Parameters

• anchor_list (list[Tensor]) – anchors of each image.

• valid_flag_list (list[Tensor]) – Valid flags of each image.

• num_level_anchors_list (list[Tensor]) – Number of anchors of each scale level of all image.

• cls_score_list (list[Tensor]) – Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_pred_list (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• unmap_outputs (bool) – Whether to map outputs back to the original set of anchors.

Returns

A tuple of reg targets components.

Return type

tuple[Tensor]

init_weights() → None

Initialize weights of the head.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], iou_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_base_priors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_base_priors * 4, H, W)

• iou_preds (list[Tensor]) – Score factor for all scale level, each is a 4D-tensor, has shape (batch_size, 1, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_cls_by_feat_single(cls_score: Tensor, labels: Tensor, label_weights: Tensor, reweight_factor: List[float], avg_factor: float) → Tuple[Tensor]

Compute cls loss of a single scale level.

Parameters

• cls_score (Tensor) – Box scores for each scale level Has shape (N, num_base_priors * num_classes, H, W).

• labels (Tensor) – Labels of each anchors with shape (N, num_total_anchors).

• label_weights (Tensor) – Label weights of each anchor with shape (N, num_total_anchors)

• reweight_factor (List[float]) – Reweight factor for cls and reg loss.

• avg_factor (float) – Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Returns

A tuple of loss components.

Return type

Tuple[Tensor]

loss_reg_by_feat_single(anchors: Tensor, bbox_pred: Tensor, iou_pred: Tensor, labels, label_weights: Tensor, bbox_targets: Tensor, bbox_weights: Tensor, reweight_factor: List[float], avg_factor: float) → Tuple[Tensor, Tensor]

Compute reg loss of a single scale level based on the features extracted by the detection head.

Parameters

• anchors (Tensor) – Box reference for each scale level with shape (N, num_total_anchors, 4).

• bbox_pred (Tensor) – Box energies / deltas for each scale level with shape (N, num_base_priors * 4, H, W).

• iou_pred (Tensor) – Iou for a single scale level, the channel number is (N, num_base_priors * 1, H, W).

• labels (Tensor) – Labels of each anchors with shape (N, num_total_anchors).

• label_weights (Tensor) – Label weights of each anchor with shape (N, num_total_anchors)

• bbox_targets (Tensor) – BBox regression targets of each anchor with shape (N, num_total_anchors, 4).

• bbox_weights (Tensor) – BBox weights of all anchors in the image with shape (N, 4)

• reweight_factor (List[float]) – Reweight factor for cls and reg loss.

• avg_factor (float) – Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Returns

A tuple of loss components.

Return type

Tuple[Tensor, Tensor]

process_predictions_and_anchors(anchor_list: List[List[Tensor]], valid_flag_list: List[List[Tensor]], cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → tuple

Compute common vars for regression and classification targets.

Parameters

• anchor_list (List[List[Tensor]]) – anchors of each image.

• valid_flag_list (List[List[Tensor]]) – Valid flags of each image.

• cls_scores (List[Tensor]) – Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_preds (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A tuple of common loss vars.

Return type

tuple[Tensor]

class mmdet.models.dense_heads.DDQDETRHead(*args, aux_num_pos=4, **kwargs)

Head of DDQDETR: Dense Distinct Query for

End-to-End Object Detection.

Code is modified from the `official github repo

<https://github.com/jshilong/DDQ>`_.

More details can be found in the `paper

<https://arxiv.org/abs/2303.12776>`_ .

Parameters

aux_num_pos (int) – Number of positive targets assigned to a perdicted object. Defaults to 4.

forward(hidden_states: Tensor, references: List[Tensor]) → Tuple[Tensor]

Forward function.

Parameters

• hidden_states (Tensor) – Hidden states output from each decoder layer, has shape (num_decoder_layers, bs, num_queries_total, dim), where num_queries_total is the sum of num_denoising_queries, num_queries and num_dense_queries when self.training is True, else num_queries.

• references (list[Tensor]) – List of the reference from the decoder. The first reference is the init_reference (initial) and the other num_decoder_layers(6) references are inter_references (intermediate). Each reference has shape (bs, num_queries_total, 4) with the last dimension arranged as (cx, cy, w, h).

Returns

results of head containing the following tensors.

• all_layers_outputs_classes (Tensor): Outputs from the classification head, has shape (num_decoder_layers, bs, num_queries_total, cls_out_channels).

• all_layers_outputs_coords (Tensor): Sigmoid outputs from the regression head with normalized coordinate format (cx, cy, w, h), has shape (num_decoder_layers, bs, num_queries_total, 4) with the last dimension arranged as (cx, cy, w, h).

Return type

tuple[Tensor]

init_weights() → None

Initialize weights of the Deformable DETR head.

loss(hidden_states: Tensor, references: List[Tensor], enc_outputs_class: Tensor, enc_outputs_coord: Tensor, batch_data_samples: List[DetDataSample], dn_meta: Dict[str, int], aux_enc_outputs_class=None, aux_enc_outputs_coord=None) → dict

Perform forward propagation and loss calculation of the detection head on the queries of the upstream network.

Parameters

• hidden_states (Tensor) – Hidden states output from each decoder layer, has shape (num_decoder_layers, bs, num_queries_total, dim), where num_queries_total is the sum of num_denoising_queries, num_queries and num_dense_queries when self.training is True, else num_queries.

• references (list[Tensor]) – List of the reference from the decoder. The first reference is the init_reference (initial) and the other num_decoder_layers(6) references are inter_references (intermediate). Each reference has shape (bs, num_queries_total, 4) with the last dimension arranged as (cx, cy, w, h).

• enc_outputs_class (Tensor) – The top k classification score of each point on encoder feature map, has shape (bs, num_queries, cls_out_channels).

• enc_outputs_coord (Tensor) – The proposal generated from points with top k score, has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• batch_data_samples (list[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• dn_meta (Dict[str, int]) – The dictionary saves information about group collation, including ‘num_denoising_queries’ and ‘num_denoising_groups’. It will be used for split outputs of denoising and matching parts and loss calculation.

• aux_enc_outputs_class (Tensor) – The dense_topk classification score of each point on encoder feature map, has shape (bs, num_dense_queries, cls_out_channels). It is None when self.training is False.

• aux_enc_outputs_coord (Tensor) – The proposal generated from points with dense_topk score, has shape (bs, num_dense_queries, 4) with the last dimension arranged as (cx, cy, w, h). It is None when self.training is False.

Returns

A dictionary of loss components.

Return type

dict

loss_by_feat(all_layers_cls_scores: Tensor, all_layers_bbox_preds: Tensor, enc_cls_scores: Tensor, enc_bbox_preds: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], dn_meta: Dict[str, int], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Loss function.

Parameters

• all_layers_cls_scores (Tensor) – Classification scores of all decoder layers, has shape (num_decoder_layers, bs, num_queries_total, cls_out_channels).

• all_layers_bbox_preds (Tensor) – Bbox coordinates of all decoder layers. Each has shape (num_decoder_layers, bs, num_queries_total, 4) with normalized coordinate format (cx, cy, w, h).

• enc_cls_scores (Tensor) – The top k score of each point on encoder feature map, has shape (bs, num_queries, cls_out_channels).

• enc_bbox_preds (Tensor) – The proposal generated from points with top k score, has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• dn_meta (Dict[str, int]) – The dictionary saves information about group collation, including ‘num_denoising_queries’ and ‘num_denoising_groups’. It will be used for split outputs of denoising and matching parts and loss calculation.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_for_distinct_queries(all_layers_cls_scores: Tensor, all_layers_bbox_preds: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Calculate the loss of distinct queries, that is, excluding denoising and dense queries. Only select the distinct queries in decoder for loss.

Parameters

• all_layers_cls_scores (Tensor) – Classification scores of all decoder layers, has shape (num_decoder_layers, bs, num_queries, cls_out_channels).

• all_layers_bbox_preds (Tensor) – Bbox coordinates of all decoder layers. It has shape (num_decoder_layers, bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image,

• e.g. –

• size (image) –

• factor (scaling) –

• etc. –

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

predict_by_feat(layer_cls_scores: Tensor, layer_bbox_preds: Tensor, batch_img_metas: List[dict], rescale: bool = True) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Parameters

• layer_cls_scores (Tensor) – Classification scores of all decoder layers, has shape (num_decoder_layers, bs, num_queries, cls_out_channels).

• layer_bbox_preds (Tensor) – Bbox coordinates of all decoder layers. Each has shape (num_decoder_layers, bs, num_queries, 4) with normalized coordinate format (cx, cy, w, h).

• batch_img_metas (list[dict]) – Meta information of each image.

• rescale (bool, optional) – If True, return boxes in original image space. Default False.

Returns

InstanceData]: Detection results of each image after the post process.

Return type

list[obj

class mmdet.models.dense_heads.DETRHead(num_classes: int, embed_dims: int = 256, num_reg_fcs: int = 2, sync_cls_avg_factor: bool = False, loss_cls: Union[ConfigDict, dict] = {'bg_cls_weight': 0.1, 'class_weight': 1.0, 'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': False}, loss_bbox: Union[ConfigDict, dict] = {'loss_weight': 5.0, 'type': 'L1Loss'}, loss_iou: Union[ConfigDict, dict] = {'loss_weight': 2.0, 'type': 'GIoULoss'}, train_cfg: Union[ConfigDict, dict] = {'assigner': {'match_costs': [{'type': 'ClassificationCost', 'weight': 1.0}, {'type': 'BBoxL1Cost', 'weight': 5.0, 'box_format': 'xywh'}, {'type': 'IoUCost', 'iou_mode': 'giou', 'weight': 2.0}], 'type': 'HungarianAssigner'}}, test_cfg: Union[ConfigDict, dict] = {'max_per_img': 100}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Head of DETR. DETR:End-to-End Object Detection with Transformers.

More details can be found in the paper .

Parameters

• num_classes (int) – Number of categories excluding the background.

• embed_dims (int) – The dims of Transformer embedding.

• num_reg_fcs (int) – Number of fully-connected layers used in FFN, which is then used for the regression head. Defaults to 2.

• sync_cls_avg_factor (bool) – Whether to sync the avg_factor of all ranks. Default to False.

• loss_cls (ConfigDict or dict) – Config of the classification loss. Defaults to CrossEntropyLoss.

• loss_bbox (ConfigDict or dict) – Config of the regression bbox loss. Defaults to L1Loss.

• loss_iou (ConfigDict or dict) – Config of the regression iou loss. Defaults to GIoULoss.

• train_cfg (ConfigDict or dict) – Training config of transformer head.

• test_cfg (ConfigDict or dict) – Testing config of transformer head.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

forward(hidden_states: Tensor) → Tuple[Tensor]

“Forward function.

Parameters

hidden_states (Tensor) – Features from transformer decoder. If return_intermediate_dec in detr.py is True output has shape (num_decoder_layers, bs, num_queries, dim), else has shape (1, bs, num_queries, dim) which only contains the last layer outputs.

Returns

results of head containing the following tensor.

• layers_cls_scores (Tensor): Outputs from the classification head, shape (num_decoder_layers, bs, num_queries, cls_out_channels). Note cls_out_channels should include background.

• layers_bbox_preds (Tensor): Sigmoid outputs from the regression head with normalized coordinate format (cx, cy, w, h), has shape (num_decoder_layers, bs, num_queries, 4).

Return type

tuple[Tensor]

get_targets(cls_scores_list: List[Tensor], bbox_preds_list: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict]) → tuple

Compute regression and classification targets for a batch image.

Outputs from a single decoder layer of a single feature level are used.

Parameters

• cls_scores_list (list[Tensor]) – Box score logits from a single decoder layer for each image, has shape [num_queries, cls_out_channels].

• bbox_preds_list (list[Tensor]) – Sigmoid outputs from a single decoder layer for each image, with normalized coordinate (cx, cy, w, h) and shape [num_queries, 4].

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

Returns

a tuple containing the following targets.

• labels_list (list[Tensor]): Labels for all images.

• label_weights_list (list[Tensor]): Label weights for all images.

• bbox_targets_list (list[Tensor]): BBox targets for all images.

• bbox_weights_list (list[Tensor]): BBox weights for all images.

• num_total_pos (int): Number of positive samples in all images.

• num_total_neg (int): Number of negative samples in all images.

Return type

tuple

loss(hidden_states: Tensor, batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection head on the features of the upstream network.

Parameters

• hidden_states (Tensor) – Feature from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, cls_out_channels) or (num_decoder_layers, num_queries, bs, cls_out_channels).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

loss_and_predict(hidden_states: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → Tuple[dict, List[InstanceData]]

Perform forward propagation of the head, then calculate loss and predictions from the features and data samples. Over-write because img_metas are needed as inputs for bbox_head.

Parameters

• hidden_states (tuple[Tensor]) – Feature from the transformer decoder, has shape (num_decoder_layers, bs, num_queries, dim).

• batch_data_samples (list[DetDataSample]) – Each item contains the meta information of each image and corresponding annotations.

Returns

the return value is a tuple contains:

• losses: (dict[str, Tensor]): A dictionary of loss components.

• predictions (list[InstanceData]): Detection results of each image after the post process.

Return type

tuple

loss_by_feat(all_layers_cls_scores: Tensor, all_layers_bbox_preds: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

“Loss function.

Only outputs from the last feature level are used for computing losses by default.

Parameters

• all_layers_cls_scores (Tensor) – Classification outputs of each decoder layers. Each is a 4D-tensor, has shape (num_decoder_layers, bs, num_queries, cls_out_channels).

• all_layers_bbox_preds (Tensor) – Sigmoid regression outputs of each decoder layers. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and shape (num_decoder_layers, bs, num_queries, 4).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_by_feat_single(cls_scores: Tensor, bbox_preds: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict]) → Tuple[Tensor]

Loss function for outputs from a single decoder layer of a single feature level.

Parameters

• cls_scores (Tensor) – Box score logits from a single decoder layer for all images, has shape (bs, num_queries, cls_out_channels).

• bbox_preds (Tensor) – Sigmoid outputs from a single decoder layer for all images, with normalized coordinate (cx, cy, w, h) and shape (bs, num_queries, 4).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

Returns

A tuple including loss_cls, loss_box and loss_iou.

Return type

Tuple[Tensor]

predict(hidden_states: Tuple[Tensor], batch_data_samples: List[DetDataSample], rescale: bool = True) → List[InstanceData]

Perform forward propagation of the detection head and predict detection results on the features of the upstream network. Over-write because img_metas are needed as inputs for bbox_head.

Parameters

• hidden_states (tuple[Tensor]) – Multi-level features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool, optional) – Whether to rescale the results. Defaults to True.

Returns

InstanceData]: Detection results of each image after the post process.

Return type

list[obj

predict_by_feat(layer_cls_scores: Tensor, layer_bbox_preds: Tensor, batch_img_metas: List[dict], rescale: bool = True) → List[InstanceData]

Transform network outputs for a batch into bbox predictions.

Parameters

• layer_cls_scores (Tensor) – Classification outputs of the last or all decoder layer. Each is a 4D-tensor, has shape (num_decoder_layers, bs, num_queries, cls_out_channels).

• layer_bbox_preds (Tensor) – Sigmoid regression outputs of the last or all decoder layer. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and shape (num_decoder_layers, bs, num_queries, 4).

• batch_img_metas (list[dict]) – Meta information of each image.

• rescale (bool, optional) – If True, return boxes in original image space. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.dense_heads.DINOHead(*args, share_pred_layer: bool = False, num_pred_layer: int = 6, as_two_stage: bool = False, **kwargs)

Head of the DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

Code is modified from the official github repo.

More details can be found in the paper .

get_dn_targets(batch_gt_instances: List[InstanceData], batch_img_metas: dict, dn_meta: Dict[str, int]) → tuple

Get targets in denoising part for a batch of images.

Parameters

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• dn_meta (Dict[str, int]) – The dictionary saves information about group collation, including ‘num_denoising_queries’ and ‘num_denoising_groups’. It will be used for split outputs of denoising and matching parts and loss calculation.

Returns

a tuple containing the following targets.

• labels_list (list[Tensor]): Labels for all images.

• label_weights_list (list[Tensor]): Label weights for all images.

• bbox_targets_list (list[Tensor]): BBox targets for all images.

• bbox_weights_list (list[Tensor]): BBox weights for all images.

• num_total_pos (int): Number of positive samples in all images.

• num_total_neg (int): Number of negative samples in all images.

Return type

tuple

loss(hidden_states: Tensor, references: List[Tensor], enc_outputs_class: Tensor, enc_outputs_coord: Tensor, batch_data_samples: List[DetDataSample], dn_meta: Dict[str, int]) → dict

Perform forward propagation and loss calculation of the detection head on the queries of the upstream network.

Parameters

• hidden_states (Tensor) – Hidden states output from each decoder layer, has shape (num_decoder_layers, bs, num_queries_total, dim), where num_queries_total is the sum of num_denoising_queries and num_matching_queries when self.training is True, else num_matching_queries.

• references (list[Tensor]) – List of the reference from the decoder. The first reference is the init_reference (initial) and the other num_decoder_layers(6) references are inter_references (intermediate). The init_reference has shape (bs, num_queries_total, 4) and each inter_reference has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• enc_outputs_class (Tensor) – The score of each point on encode feature map, has shape (bs, num_feat_points, cls_out_channels).

• enc_outputs_coord (Tensor) – The proposal generate from the encode feature map, has shape (bs, num_feat_points, 4) with the last dimension arranged as (cx, cy, w, h).

• batch_data_samples (list[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• dn_meta (Dict[str, int]) – The dictionary saves information about group collation, including ‘num_denoising_queries’ and ‘num_denoising_groups’. It will be used for split outputs of denoising and matching parts and loss calculation.

Returns

A dictionary of loss components.

Return type

dict

loss_by_feat(all_layers_cls_scores: Tensor, all_layers_bbox_preds: Tensor, enc_cls_scores: Tensor, enc_bbox_preds: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], dn_meta: Dict[str, int], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Loss function.

Parameters

• all_layers_cls_scores (Tensor) – Classification scores of all decoder layers, has shape (num_decoder_layers, bs, num_queries_total, cls_out_channels), where num_queries_total is the sum of num_denoising_queries and num_matching_queries.

• all_layers_bbox_preds (Tensor) – Regression outputs of all decoder layers. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and has shape (num_decoder_layers, bs, num_queries_total, 4).

• enc_cls_scores (Tensor) – The score of each point on encode feature map, has shape (bs, num_feat_points, cls_out_channels).

• enc_bbox_preds (Tensor) – The proposal generate from the encode feature map, has shape (bs, num_feat_points, 4) with the last dimension arranged as (cx, cy, w, h).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• dn_meta (Dict[str, int]) – The dictionary saves information about group collation, including ‘num_denoising_queries’ and ‘num_denoising_groups’. It will be used for split outputs of denoising and matching parts and loss calculation.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_dn(all_layers_denoising_cls_scores: Tensor, all_layers_denoising_bbox_preds: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], dn_meta: Dict[str, int]) → Tuple[List[Tensor]]

Calculate denoising loss.

Parameters

• all_layers_denoising_cls_scores (Tensor) – Classification scores of all decoder layers in denoising part, has shape ( num_decoder_layers, bs, num_denoising_queries, cls_out_channels).

• all_layers_denoising_bbox_preds (Tensor) – Regression outputs of all decoder layers in denoising part. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and has shape (num_decoder_layers, bs, num_denoising_queries, 4).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• dn_meta (Dict[str, int]) – The dictionary saves information about group collation, including ‘num_denoising_queries’ and ‘num_denoising_groups’. It will be used for split outputs of denoising and matching parts and loss calculation.

Returns

The loss_dn_cls, loss_dn_bbox, and loss_dn_iou of each decoder layers.

Return type

Tuple[List[Tensor]]

static split_outputs(all_layers_cls_scores: Tensor, all_layers_bbox_preds: Tensor, dn_meta: Dict[str, int]) → Tuple[Tensor]

Split outputs of the denoising part and the matching part.

For the total outputs of num_queries_total length, the former num_denoising_queries outputs are from denoising queries, and the rest num_matching_queries ones are from matching queries, where num_queries_total is the sum of num_denoising_queries and num_matching_queries.

Parameters

• all_layers_cls_scores (Tensor) – Classification scores of all decoder layers, has shape (num_decoder_layers, bs, num_queries_total, cls_out_channels).

• all_layers_bbox_preds (Tensor) – Regression outputs of all decoder layers. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and has shape (num_decoder_layers, bs, num_queries_total, 4).

• dn_meta (Dict[str, int]) – The dictionary saves information about group collation, including ‘num_denoising_queries’ and ‘num_denoising_groups’.

Returns

a tuple containing the following outputs.

• all_layers_matching_cls_scores (Tensor): Classification scores of all decoder layers in matching part, has shape (num_decoder_layers, bs, num_matching_queries, cls_out_channels).

• all_layers_matching_bbox_preds (Tensor): Regression outputs of all decoder layers in matching part. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and has shape (num_decoder_layers, bs, num_matching_queries, 4).

• all_layers_denoising_cls_scores (Tensor): Classification scores of all decoder layers in denoising part, has shape (num_decoder_layers, bs, num_denoising_queries, cls_out_channels).

• all_layers_denoising_bbox_preds (Tensor): Regression outputs of all decoder layers in denoising part. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and has shape (num_decoder_layers, bs, num_denoising_queries, 4).

Return type

Tuple[Tensor]

class mmdet.models.dense_heads.DecoupledSOLOHead(*args, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = [{'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}, {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01, 'override': {'name': 'conv_mask_list_x'}}, {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01, 'override': {'name': 'conv_mask_list_y'}}, {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01, 'override': {'name': 'conv_cls'}}], **kwargs)

Decoupled SOLO mask head used in `SOLO: Segmenting Objects by Locations.

<https://arxiv.org/abs/1912.04488>`_

Parameters

init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(x: Tuple[Tensor]) → Tuple

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of classification scores and mask prediction.

• mlvl_mask_preds_x (list[Tensor]): Multi-level mask prediction from x branch. Each element in the list has shape (batch_size, num_grids ,h ,w).

• mlvl_mask_preds_y (list[Tensor]): Multi-level mask prediction from y branch. Each element in the list has shape (batch_size, num_grids ,h ,w).

• mlvl_cls_preds (list[Tensor]): Multi-level scores. Each element in the list has shape (batch_size, num_classes, num_grids ,num_grids).

Return type

tuple

loss_by_feat(mlvl_mask_preds_x: List[Tensor], mlvl_mask_preds_y: List[Tensor], mlvl_cls_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], **kwargs) → dict

Calculate the loss based on the features extracted by the mask head.

Parameters

• mlvl_mask_preds_x (list[Tensor]) – Multi-level mask prediction from x branch. Each element in the list has shape (batch_size, num_grids ,h ,w).

• mlvl_mask_preds_y (list[Tensor]) – Multi-level mask prediction from y branch. Each element in the list has shape (batch_size, num_grids ,h ,w).

• mlvl_cls_preds (list[Tensor]) – Multi-level scores. Each element in the list has shape (batch_size, num_classes, num_grids ,num_grids).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, masks, and labels attributes.

• batch_img_metas (list[dict]) – Meta information of multiple images.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

predict_by_feat(mlvl_mask_preds_x: List[Tensor], mlvl_mask_preds_y: List[Tensor], mlvl_cls_scores: List[Tensor], batch_img_metas: List[dict], **kwargs) → List[InstanceData]

Transform a batch of output features extracted from the head into mask results.

Parameters

• mlvl_mask_preds_x (list[Tensor]) – Multi-level mask prediction from x branch. Each element in the list has shape (batch_size, num_grids ,h ,w).

• mlvl_mask_preds_y (list[Tensor]) – Multi-level mask prediction from y branch. Each element in the list has shape (batch_size, num_grids ,h ,w).

• mlvl_cls_scores (list[Tensor]) – Multi-level scores. Each element in the list has shape (batch_size, num_classes ,num_grids ,num_grids).

• batch_img_metas (list[dict]) – Meta information of all images.

Returns

Processed results of multiple images.Each InstanceData usually contains following keys.

• scores (Tensor): Classification scores, has shape (num_instance,).

• labels (Tensor): Has shape (num_instances,).

• masks (Tensor): Processed mask results, has shape (num_instances, h, w).

Return type

list[InstanceData]

class mmdet.models.dense_heads.DecoupledSOLOLightHead(*args, dcn_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = [{'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}, {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01, 'override': {'name': 'conv_mask_list_x'}}, {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01, 'override': {'name': 'conv_mask_list_y'}}, {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01, 'override': {'name': 'conv_cls'}}], **kwargs)

Decoupled Light SOLO mask head used in SOLO: Segmenting Objects by Locations

Parameters

• with_dcn (bool) – Whether use dcn in mask_convs and cls_convs, Defaults to False.

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(x: Tuple[Tensor]) → Tuple

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of classification scores and mask prediction.

• mlvl_mask_preds_x (list[Tensor]): Multi-level mask prediction from x branch. Each element in the list has shape (batch_size, num_grids ,h ,w).

• mlvl_mask_preds_y (list[Tensor]): Multi-level mask prediction from y branch. Each element in the list has shape (batch_size, num_grids ,h ,w).

• mlvl_cls_preds (list[Tensor]): Multi-level scores. Each element in the list has shape (batch_size, num_classes, num_grids ,num_grids).

Return type

tuple

class mmdet.models.dense_heads.DeformableDETRHead(*args, share_pred_layer: bool = False, num_pred_layer: int = 6, as_two_stage: bool = False, **kwargs)

Head of DeformDETR: Deformable DETR: Deformable Transformers for End-to-End Object Detection.

Code is modified from the official github repo.

More details can be found in the paper .

Parameters

• share_pred_layer (bool) – Whether to share parameters for all the prediction layers. Defaults to False.

• num_pred_layer (int) – The number of the prediction layers. Defaults to 6.

• as_two_stage (bool, optional) – Whether to generate the proposal from the outputs of encoder. Defaults to False.

forward(hidden_states: Tensor, references: List[Tensor]) → Tuple[Tensor, Tensor]

Forward function.

Parameters

• hidden_states (Tensor) – Hidden states output from each decoder layer, has shape (num_decoder_layers, bs, num_queries, dim).

• references (list[Tensor]) – List of the reference from the decoder. The first reference is the init_reference (initial) and the other num_decoder_layers(6) references are inter_references (intermediate). The init_reference has shape (bs, num_queries, 4) when as_two_stage of the detector is True, otherwise (bs, num_queries, 2). Each inter_reference has shape (bs, num_queries, 4) when with_box_refine of the detector is True, otherwise (bs, num_queries, 2). The coordinates are arranged as (cx, cy) when the last dimension is 2, and (cx, cy, w, h) when it is 4.

Returns

results of head containing the following tensor.

• all_layers_outputs_classes (Tensor): Outputs from the classification head, has shape (num_decoder_layers, bs, num_queries, cls_out_channels).

• all_layers_outputs_coords (Tensor): Sigmoid outputs from the regression head with normalized coordinate format (cx, cy, w, h), has shape (num_decoder_layers, bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

Return type

tuple[Tensor]

init_weights() → None

Initialize weights of the Deformable DETR head.

loss(hidden_states: Tensor, references: List[Tensor], enc_outputs_class: Tensor, enc_outputs_coord: Tensor, batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection head on the queries of the upstream network.

Parameters

• hidden_states (Tensor) – Hidden states output from each decoder layer, has shape (num_decoder_layers, num_queries, bs, dim).

• references (list[Tensor]) – List of the reference from the decoder. The first reference is the init_reference (initial) and the other num_decoder_layers(6) references are inter_references (intermediate). The init_reference has shape (bs, num_queries, 4) when as_two_stage of the detector is True, otherwise (bs, num_queries, 2). Each inter_reference has shape (bs, num_queries, 4) when with_box_refine of the detector is True, otherwise (bs, num_queries, 2). The coordinates are arranged as (cx, cy) when the last dimension is 2, and (cx, cy, w, h) when it is 4.

• enc_outputs_class (Tensor) – The score of each point on encode feature map, has shape (bs, num_feat_points, cls_out_channels). Only when as_two_stage is True it would be passed in, otherwise it would be None.

• enc_outputs_coord (Tensor) – The proposal generate from the encode feature map, has shape (bs, num_feat_points, 4) with the last dimension arranged as (cx, cy, w, h). Only when as_two_stage is True it would be passed in, otherwise it would be None.

• batch_data_samples (list[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

loss_by_feat(all_layers_cls_scores: Tensor, all_layers_bbox_preds: Tensor, enc_cls_scores: Tensor, enc_bbox_preds: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Loss function.

Parameters

• all_layers_cls_scores (Tensor) – Classification scores of all decoder layers, has shape (num_decoder_layers, bs, num_queries, cls_out_channels).

• all_layers_bbox_preds (Tensor) – Regression outputs of all decoder layers. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and has shape (num_decoder_layers, bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• enc_cls_scores (Tensor) – The score of each point on encode feature map, has shape (bs, num_feat_points, cls_out_channels). Only when as_two_stage is True it would be passes in, otherwise, it would be None.

• enc_bbox_preds (Tensor) – The proposal generate from the encode feature map, has shape (bs, num_feat_points, 4) with the last dimension arranged as (cx, cy, w, h). Only when as_two_stage is True it would be passed in, otherwise it would be None.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

predict(hidden_states: Tensor, references: List[Tensor], batch_data_samples: List[DetDataSample], rescale: bool = True) → List[InstanceData]

Perform forward propagation and loss calculation of the detection head on the queries of the upstream network.

Parameters

• hidden_states (Tensor) – Hidden states output from each decoder layer, has shape (num_decoder_layers, num_queries, bs, dim).

• references (list[Tensor]) – List of the reference from the decoder. The first reference is the init_reference (initial) and the other num_decoder_layers(6) references are inter_references (intermediate). The init_reference has shape (bs, num_queries, 4) when as_two_stage of the detector is True, otherwise (bs, num_queries, 2). Each inter_reference has shape (bs, num_queries, 4) when with_box_refine of the detector is True, otherwise (bs, num_queries, 2). The coordinates are arranged as (cx, cy) when the last dimension is 2, and (cx, cy, w, h) when it is 4.

• batch_data_samples (list[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool, optional) – If True, return boxes in original image space. Defaults to True.

Returns

InstanceData]: Detection results of each image after the post process.

Return type

list[obj

predict_by_feat(all_layers_cls_scores: Tensor, all_layers_bbox_preds: Tensor, batch_img_metas: List[Dict], rescale: bool = False) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Parameters

• all_layers_cls_scores (Tensor) – Classification scores of all decoder layers, has shape (num_decoder_layers, bs, num_queries, cls_out_channels).

• all_layers_bbox_preds (Tensor) – Regression outputs of all decoder layers. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and shape (num_decoder_layers, bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• batch_img_metas (list[dict]) – Meta information of each image.

• rescale (bool, optional) – If True, return boxes in original image space. Default False.

Returns

InstanceData]: Detection results of each image after the post process.

Return type

list[obj

class mmdet.models.dense_heads.EmbeddingRPNHead(num_proposals: int = 100, proposal_feature_channel: int = 256, init_cfg: Optional[Union[ConfigDict, dict]] = None, **kwargs)

RPNHead in the Sparse R-CNN .

Unlike traditional RPNHead, this module does not need FPN input, but just decode init_proposal_bboxes and expand the first dimension of init_proposal_bboxes and init_proposal_features to the batch_size.

Parameters

• num_proposals (int) – Number of init_proposals. Defaults to 100.

• proposal_feature_channel (int) – Channel number of init_proposal_feature. Defaults to 256.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict]) – Initialization config dict. Defaults to None.

init_weights() → None

Initialize the init_proposal_bboxes as normalized.

[c_x, c_y, w, h], and we initialize it to the size of the entire image.

loss(*args, **kwargs)

Perform forward propagation and loss calculation of the detection head on the features of the upstream network.

loss_and_predict(x: List[Tensor], batch_data_samples: List[DetDataSample], **kwargs) → tuple

Perform forward propagation of the head, then calculate loss and predictions from the features and data samples.

predict(x: List[Tensor], batch_data_samples: List[DetDataSample], **kwargs) → List[InstanceData]

Perform forward propagation of the detection head and predict detection results on the features of the upstream network.

class mmdet.models.dense_heads.FCOSHead(num_classes: int, in_channels: int, regress_ranges: Sequence[Tuple[int, int]] = ((-1, 64), (64, 128), (128, 256), (256, 512), (512, 100000000.0)), center_sampling: bool = False, center_sample_radius: float = 1.5, norm_on_bbox: bool = False, centerness_on_reg: bool = False, loss_cls: Union[ConfigDict, dict] = {'alpha': 0.25, 'gamma': 2.0, 'loss_weight': 1.0, 'type': 'FocalLoss', 'use_sigmoid': True}, loss_bbox: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'IoULoss'}, loss_centerness: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 32, 'requires_grad': True, 'type': 'GN'}, cls_predictor_cfg=None, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'conv_cls', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'}, **kwargs)

Anchor-free head used in FCOS.

The FCOS head does not use anchor boxes. Instead bounding boxes are predicted at each pixel and a centerness measure is used to suppress low-quality predictions. Here norm_on_bbox, centerness_on_reg, dcn_on_last_conv are training tricks used in official repo, which will bring remarkable mAP gains of up to 4.9. Please see https://github.com/tianzhi0549/FCOS for more detail.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• strides (Sequence[int] or Sequence[Tuple[int, int]]) – Strides of points in multiple feature levels. Defaults to (4, 8, 16, 32, 64).

• regress_ranges (Sequence[Tuple[int, int]]) – Regress range of multiple level points.

• center_sampling (bool) – If true, use center sampling. Defaults to False.

• center_sample_radius (float) – Radius of center sampling. Defaults to 1.5.

• norm_on_bbox (bool) – If true, normalize the regression targets with FPN strides. Defaults to False.

• centerness_on_reg (bool) – If true, position centerness on the regress branch. Please refer to https://github.com/tianzhi0549/FCOS/issues/89#issuecomment-516877042. Defaults to False.

• conv_bias (bool or str) – If specified as auto, it will be decided by the norm_cfg. Bias of conv will be set as True if norm_cfg is None, otherwise False. Defaults to “auto”.

• loss_cls (ConfigDict or dict) – Config of classification loss.

• loss_bbox (ConfigDict or dict) – Config of localization loss.

• loss_centerness (ConfigDict, or dict) – Config of centerness loss.

• norm_cfg (ConfigDict or dict) – dictionary to construct and config norm layer. Defaults to norm_cfg=dict(type='GN', num_groups=32, requires_grad=True).

• cls_predictor_cfg (ConfigDict or dict) – dictionary to construct and config conv_cls. Defaults to None.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict]) – Initialization config dict.

Example

>>> self = FCOSHead(11, 7)
>>> feats = [torch.rand(1, 7, s, s) for s in [4, 8, 16, 32, 64]]
>>> cls_score, bbox_pred, centerness = self.forward(feats)
>>> assert len(cls_score) == len(self.scales)

centerness_target(pos_bbox_targets: Tensor) → Tensor

Compute centerness targets.

Parameters

pos_bbox_targets (Tensor) – BBox targets of positive bboxes in shape (num_pos, 4)

Returns

Centerness target.

Return type

Tensor

forward(x: Tuple[Tensor]) → Tuple[List[Tensor], List[Tensor], List[Tensor]]

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of each level outputs.

• cls_scores (list[Tensor]): Box scores for each scale level, each is a 4D-tensor, the channel number is num_points * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

• centernesses (list[Tensor]): centerness for each scale level, each is a 4D-tensor, the channel number is num_points * 1.

Return type

tuple

forward_single(x: Tensor, scale: Scale, stride: int) → Tuple[Tensor, Tensor, Tensor]

Forward features of a single scale level.

Parameters

• x (Tensor) – FPN feature maps of the specified stride.

• scale (mmcv.cnn.Scale) – Learnable scale module to resize the bbox prediction.

• stride (int) – The corresponding stride for feature maps, only used to normalize the bbox prediction when self.norm_on_bbox is True.

Returns

scores for each class, bbox predictions and centerness predictions of input feature maps.

Return type

tuple

get_targets(points: List[Tensor], batch_gt_instances: List[InstanceData]) → Tuple[List[Tensor], List[Tensor]]

Compute regression, classification and centerness targets for points in multiple images.

Parameters

• points (list[Tensor]) – Points of each fpn level, each has shape (num_points, 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

Returns

Targets of each level.

• concat_lvl_labels (list[Tensor]): Labels of each level.

• concat_lvl_bbox_targets (list[Tensor]): BBox targets of each level.

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], centernesses: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level, each is a 4D-tensor, the channel number is num_points * num_classes.

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

• centernesses (list[Tensor]) – centerness for each scale level, each is a 4D-tensor, the channel number is num_points * 1.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.FSAFHead(*args, score_threshold: Optional[float] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, **kwargs)

Anchor-free head used in FSAF.

The head contains two subnetworks. The first classifies anchor boxes and the second regresses deltas for the anchors (num_anchors is 1 for anchor- free methods)

Parameters

• *args – Same as its base class in RetinaHead

• score_threshold (float, optional) – The score_threshold to calculate positive recall. If given, prediction scores lower than this value is counted as incorrect prediction. Defaults to None.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict]) – Initialization config dict.

• **kwargs – Same as its base class in RetinaHead

Example

>>> import torch
>>> self = FSAFHead(11, 7)
>>> x = torch.rand(1, 7, 32, 32)
>>> cls_score, bbox_pred = self.forward_single(x)
>>> # Each anchor predicts a score for each class except background
>>> cls_per_anchor = cls_score.shape[1] / self.num_anchors
>>> box_per_anchor = bbox_pred.shape[1] / self.num_anchors
>>> assert cls_per_anchor == self.num_classes
>>> assert box_per_anchor == 4

calculate_pos_recall(cls_scores: List[Tensor], labels_list: List[Tensor], pos_inds: List[Tensor]) → Tensor

Calculate positive recall with score threshold.

Parameters

• cls_scores (list[Tensor]) – Classification scores at all fpn levels. Each tensor is in shape (N, num_classes * num_anchors, H, W)

• labels_list (list[Tensor]) – The label that each anchor is assigned to. Shape (N * H * W * num_anchors, )

• pos_inds (list[Tensor]) – List of bool tensors indicating whether the anchor is assigned to a positive label. Shape (N * H * W * num_anchors, )

Returns

A single float number indicating the positive recall.

Return type

Tensor

collect_loss_level_single(cls_loss: Tensor, reg_loss: Tensor, assigned_gt_inds: Tensor, labels_seq: Tensor) → Tensor

Get the average loss in each FPN level w.r.t. each gt label.

Parameters

• cls_loss (Tensor) – Classification loss of each feature map pixel, shape (num_anchor, num_class)

• reg_loss (Tensor) – Regression loss of each feature map pixel, shape (num_anchor, 4)

• assigned_gt_inds (Tensor) – It indicates which gt the prior is assigned to (0-based, -1: no assignment). shape (num_anchor),

• labels_seq – The rank of labels. shape (num_gt)

Returns

shape (num_gt), average loss of each gt in this level

Return type

Tensor

forward_single(x: Tensor) → Tuple[Tensor, Tensor]

Forward feature map of a single scale level.

Parameters

x (Tensor) – Feature map of a single scale level.

Returns

• cls_score (Tensor): Box scores for each scale level Has shape (N, num_points * num_classes, H, W).

• bbox_pred (Tensor): Box energies / deltas for each scale level with shape (N, num_points * 4, H, W).

Return type

tuple[Tensor, Tensor]

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Compute loss of the head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_points * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_points * 4, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

reweight_loss_single(cls_loss: Tensor, reg_loss: Tensor, assigned_gt_inds: Tensor, labels: Tensor, level: int, min_levels: Tensor) → tuple

Reweight loss values at each level.

Reassign loss values at each level by masking those where the pre-calculated loss is too large. Then return the reduced losses.

Parameters

• cls_loss (Tensor) – Element-wise classification loss. Shape: (num_anchors, num_classes)

• reg_loss (Tensor) – Element-wise regression loss. Shape: (num_anchors, 4)

• assigned_gt_inds (Tensor) – The gt indices that each anchor bbox is assigned to. -1 denotes a negative anchor, otherwise it is the gt index (0-based). Shape: (num_anchors, ),

• labels (Tensor) – Label assigned to anchors. Shape: (num_anchors, ).

• level (int) – The current level index in the pyramid (0-4 for RetinaNet)

• min_levels (Tensor) – The best-matching level for each gt. Shape: (num_gts, ),

Returns

• cls_loss: Reduced corrected classification loss. Scalar.

• reg_loss: Reduced corrected regression loss. Scalar.

• pos_flags (Tensor): Corrected bool tensor indicating the final positive anchors. Shape: (num_anchors, ).

Return type

tuple

class mmdet.models.dense_heads.FeatureAdaption(in_channels: int, out_channels: int, kernel_size: int = 3, deform_groups: int = 4, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'name': 'conv_adaption', 'std': 0.01, 'type': 'Normal'}, 'std': 0.1, 'type': 'Normal'})

Feature Adaption Module.

Feature Adaption Module is implemented based on DCN v1. It uses anchor shape prediction rather than feature map to predict offsets of deform conv layer.

Parameters

• in_channels (int) – Number of channels in the input feature map.

• out_channels (int) – Number of channels in the output feature map.

• kernel_size (int) – Deformable conv kernel size. Defaults to 3.

• deform_groups (int) – Deformable conv group size. Defaults to 4.

• init_cfg (ConfigDict or list[ConfigDict] or dict or list[dict], optional) – Initialization config dict.

forward(x: Tensor, shape: Tensor) → Tensor

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.dense_heads.FoveaHead(num_classes: int, in_channels: int, base_edge_list: List[int] = (16, 32, 64, 128, 256), scale_ranges: List[tuple] = ((8, 32), (16, 64), (32, 128), (64, 256), (128, 512)), sigma: float = 0.4, with_deform: bool = False, deform_groups: int = 4, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = {'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'conv_cls', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'}, **kwargs)

Detection Head of `FoveaBox: Beyond Anchor-based Object Detector.

<https://arxiv.org/abs/1904.03797>`_.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• base_edge_list (list[int]) – List of edges.

• scale_ranges (list[tuple]) – Range of scales.

• sigma (float) – Super parameter of FoveaHead.

• with_deform (bool) – Whether use deform conv.

• deform_groups (int) – Deformable conv group size.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict.

forward_single(x: Tensor) → Tuple[Tensor, Tensor]

Forward features of a single scale level.

Parameters

x (Tensor) – FPN feature maps of the specified stride.

Returns

scores for each class and bbox predictions of input feature maps.

Return type

tuple

get_targets(batch_gt_instances: List[InstanceData], featmap_sizes: List[tuple], priors_list: List[Tensor]) → Tuple[List[Tensor], List[Tensor]]

Compute regression and classification for priors in multiple images.

Parameters

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• featmap_sizes (list[tuple]) – Size tuple of feature maps.

• priors_list (list[Tensor]) – Priors list of each fpn level, each has shape (num_priors, 2).

Returns

Targets of each level.

• flatten_labels (list[Tensor]): Labels of each level.

• flatten_bbox_targets (list[Tensor]): BBox targets of each level.

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level, each is a 4D-tensor, the channel number is num_priors * num_classes.

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is num_priors * 4.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.FreeAnchorRetinaHead(num_classes: int, in_channels: int, stacked_convs: int = 4, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, pre_anchor_topk: int = 50, bbox_thr: float = 0.6, gamma: float = 2.0, alpha: float = 0.5, **kwargs)

FreeAnchor RetinaHead used in https://arxiv.org/abs/1909.02466.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• stacked_convs (int) – Number of conv layers in cls and reg tower. Defaults to 4.

• conv_cfg (ConfigDict or dict, optional) – dictionary to construct and config conv layer. Defaults to None.

• norm_cfg (ConfigDict or dict, optional) – dictionary to construct and config norm layer. Defaults to norm_cfg=dict(type=’GN’, num_groups=32, requires_grad=True).

• pre_anchor_topk (int) – Number of boxes that be token in each bag. Defaults to 50

• bbox_thr (float) – The threshold of the saturated linear function. It is usually the same with the IoU threshold used in NMS. Defaults to 0.6.

• gamma (float) – Gamma parameter in focal loss. Defaults to 2.0.

• alpha (float) – Alpha parameter in focal loss. Defaults to 0.5.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level has shape (N, num_anchors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict

negative_bag_loss(cls_prob: Tensor, box_prob: Tensor) → Tensor

Compute negative bag loss.

\(FL((1 - P_{a_{j} \in A_{+}}) * (1 - P_{j}^{bg}))\).

\(P_{a_{j} \in A_{+}}\): Box_probability of matched samples.

\(P_{j}^{bg}\): Classification probability of negative samples.

Parameters

• cls_prob (Tensor) – Classification probability, in shape (num_img, num_anchors, num_classes).

• box_prob (Tensor) – Box probability, in shape (num_img, num_anchors, num_classes).

Returns

Negative bag loss in shape (num_img, num_anchors, num_classes).

Return type

Tensor

positive_bag_loss(matched_cls_prob: Tensor, matched_box_prob: Tensor) → Tensor

Compute positive bag loss.

\(-log( Mean-max(P_{ij}^{cls} * P_{ij}^{loc}) )\).

\(P_{ij}^{cls}\): matched_cls_prob, classification probability of matched samples.

\(P_{ij}^{loc}\): matched_box_prob, box probability of matched samples.

Parameters

• matched_cls_prob (Tensor) – Classification probability of matched samples in shape (num_gt, pre_anchor_topk).

• matched_box_prob (Tensor) – BBox probability of matched samples, in shape (num_gt, pre_anchor_topk).

Returns

Positive bag loss in shape (num_gt,).

Return type

Tensor

positive_loss_single(cls_prob: Tensor, bbox_pred: Tensor, flat_anchors: Tensor, gt_instances: InstanceData) → tuple

Compute positive loss.

Parameters

• cls_prob (Tensor) – Classification probability of shape (num_anchors, num_classes).

• bbox_pred (Tensor) – Box probability of shape (num_anchors, 4).

• flat_anchors (Tensor) – Multi-level anchors of the image, which are concatenated into a single tensor of shape (num_anchors, 4)

• gt_instances (InstanceData) – Ground truth of instance annotations. It should includes bboxes and labels attributes.

Returns

• box_prob (Tensor): Box probability of shape (num_anchors, 4).

• positive_loss (Tensor): Positive loss of shape (num_pos, ).

• num_pos (int): positive samples indexes.

Return type

tuple

class mmdet.models.dense_heads.GARPNHead(in_channels: int, num_classes: int = 1, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'conv_loc', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'}, **kwargs)

Guided-Anchor-based RPN head.

forward_single(x: Tensor) → Tuple[Tensor]

Forward feature of a single scale level.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], shape_preds: List[Tensor], loc_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level has shape (N, num_anchors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• shape_preds (list[Tensor]) – shape predictions for each scale level with shape (N, 1, H, W).

• loc_preds (list[Tensor]) – location predictions for each scale level with shape (N, num_anchors * 2, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict

class mmdet.models.dense_heads.GARetinaHead(num_classes: int, in_channels: int, stacked_convs: int = 4, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, **kwargs)

Guided-Anchor-based RetinaNet head.

forward_single(x: Tensor) → Tuple[Tensor]

Forward feature map of a single scale level.

class mmdet.models.dense_heads.GFLHead(num_classes: int, in_channels: int, stacked_convs: int = 4, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 32, 'requires_grad': True, 'type': 'GN'}, loss_dfl: Union[ConfigDict, dict] = {'loss_weight': 0.25, 'type': 'DistributionFocalLoss'}, bbox_coder: Union[ConfigDict, dict] = {'type': 'DistancePointBBoxCoder'}, reg_max: int = 16, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'gfl_cls', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'}, **kwargs)

Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection.

GFL head structure is similar with ATSS, however GFL uses 1) joint representation for classification and localization quality, and 2) flexible General distribution for bounding box locations, which are supervised by Quality Focal Loss (QFL) and Distribution Focal Loss (DFL), respectively

https://arxiv.org/abs/2006.04388

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• stacked_convs (int) – Number of conv layers in cls and reg tower. Defaults to 4.

• conv_cfg (ConfigDict or dict, optional) – dictionary to construct and config conv layer. Defaults to None.

• norm_cfg (ConfigDict or dict) – dictionary to construct and config norm layer. Default: dict(type=’GN’, num_groups=32, requires_grad=True).

• loss_qfl (ConfigDict or dict) – Config of Quality Focal Loss (QFL).

• bbox_coder (ConfigDict or dict) – Config of bbox coder. Defaults to ‘DistancePointBBoxCoder’.

• reg_max (int) – Max value of integral set :math: {0, ..., reg_max} in QFL setting. Defaults to 16.

:param init_cfg (ConfigDict or dict or list[dict] or: list[ConfigDict]): Initialization config dict.

Example

>>> self = GFLHead(11, 7)
>>> feats = [torch.rand(1, 7, s, s) for s in [4, 8, 16, 32, 64]]
>>> cls_quality_score, bbox_pred = self.forward(feats)
>>> assert len(cls_quality_score) == len(self.scales)

anchor_center(anchors: Tensor) → Tensor

Get anchor centers from anchors.

Parameters

anchors (Tensor) – Anchor list with shape (N, 4), xyxy format.

Returns

Anchor centers with shape (N, 2), xy format.

Return type

Tensor

forward(x: Tuple[Tensor]) → Tuple[List[Tensor]]

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually a tuple of classification scores and bbox prediction

• cls_scores (list[Tensor]): Classification and quality (IoU) joint scores for all scale levels, each is a 4D-tensor, the channel number is num_classes.

• bbox_preds (list[Tensor]): Box distribution logits for all scale levels, each is a 4D-tensor, the channel number is 4*(n+1), n is max value of integral set.

Return type

tuple

forward_single(x: Tensor, scale: Scale) → Sequence[Tensor]

Forward feature of a single scale level.

Parameters

• x (Tensor) – Features of a single scale level.

• ( (scale) – obj: mmcv.cnn.Scale): Learnable scale module to resize the bbox prediction.

Returns

• cls_score (Tensor): Cls and quality joint scores for a single scale level the channel number is num_classes.

• bbox_pred (Tensor): Box distribution logits for a single scale level, the channel number is 4*(n+1), n is max value of integral set.

Return type

tuple

get_num_level_anchors_inside(num_level_anchors: List[int], inside_flags: Tensor) → List[int]

Get the number of valid anchors in every level.

get_targets(anchor_list: List[Tensor], valid_flag_list: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs=True) → tuple

Get targets for GFL head.

This method is almost the same as AnchorHead.get_targets(). Besides returning the targets as the parent method does, it also returns the anchors as the first element of the returned tuple.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Cls and quality scores for each scale level has shape (N, num_classes, H, W).

• bbox_preds (list[Tensor]) – Box distribution logits for each scale level with shape (N, 4*(n+1), H, W), n is max value of integral set.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_by_feat_single(anchors: Tensor, cls_score: Tensor, bbox_pred: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, stride: Tuple[int], avg_factor: int) → dict

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• anchors (Tensor) – Box reference for each scale level with shape (N, num_total_anchors, 4).

• cls_score (Tensor) – Cls and quality joint scores for each scale level has shape (N, num_classes, H, W).

• bbox_pred (Tensor) – Box distribution logits for each scale level with shape (N, 4*(n+1), H, W), n is max value of integral set.

• labels (Tensor) – Labels of each anchors with shape (N, num_total_anchors).

• label_weights (Tensor) – Label weights of each anchor with shape (N, num_total_anchors)

• bbox_targets (Tensor) – BBox regression targets of each anchor with shape (N, num_total_anchors, 4).

• stride (Tuple[int]) – Stride in this scale level.

• avg_factor (int) – Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.GroundingDINOHead(contrastive_cfg={'max_text_len': 256}, **kwargs)

Head of the Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.

Parameters

contrastive_cfg (dict, optional) – Contrastive config that contains keys like max_text_len. Defaults to dict(max_text_len=256).

forward(hidden_states: Tensor, references: List[Tensor], memory_text: Tensor, text_token_mask: Tensor) → Tuple[Tensor]

Forward function.

Parameters

• hidden_states (Tensor) – Hidden states output from each decoder layer, has shape (num_decoder_layers, bs, num_queries, dim).

• references (List[Tensor]) – List of the reference from the decoder. The first reference is the init_reference (initial) and the other num_decoder_layers(6) references are inter_references (intermediate). The init_reference has shape (bs, num_queries, 4) when as_two_stage of the detector is True, otherwise (bs, num_queries, 2). Each inter_reference has shape (bs, num_queries, 4) when with_box_refine of the detector is True, otherwise (bs, num_queries, 2). The coordinates are arranged as (cx, cy) when the last dimension is 2, and (cx, cy, w, h) when it is 4.

• memory_text (Tensor) – Memory text. It has shape (bs, len_text, text_embed_dims).

• text_token_mask (Tensor) – Text token mask. It has shape (bs, len_text).

Returns

results of head containing the following tensor.

• all_layers_outputs_classes (Tensor): Outputs from the classification head, has shape (num_decoder_layers, bs, num_queries, cls_out_channels).

• all_layers_outputs_coords (Tensor): Sigmoid outputs from the regression head with normalized coordinate format (cx, cy, w, h), has shape (num_decoder_layers, bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

Return type

tuple[Tensor]

init_weights() → None

Initialize weights of the Deformable DETR head.

loss(hidden_states: Tensor, references: List[Tensor], memory_text: Tensor, text_token_mask: Tensor, enc_outputs_class: Tensor, enc_outputs_coord: Tensor, batch_data_samples: List[DetDataSample], dn_meta: Dict[str, int]) → dict

Perform forward propagation and loss calculation of the detection head on the queries of the upstream network.

Parameters

• hidden_states (Tensor) – Hidden states output from each decoder layer, has shape (num_decoder_layers, bs, num_queries_total, dim), where num_queries_total is the sum of num_denoising_queries and num_matching_queries when self.training is True, else num_matching_queries.

• references (list[Tensor]) – List of the reference from the decoder. The first reference is the init_reference (initial) and the other num_decoder_layers(6) references are inter_references (intermediate). The init_reference has shape (bs, num_queries_total, 4) and each inter_reference has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• memory_text (Tensor) – Memory text. It has shape (bs, len_text, text_embed_dims).

• enc_outputs_class (Tensor) – The score of each point on encode feature map, has shape (bs, num_feat_points, cls_out_channels).

• enc_outputs_coord (Tensor) – The proposal generate from the encode feature map, has shape (bs, num_feat_points, 4) with the last dimension arranged as (cx, cy, w, h).

• batch_data_samples (list[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• dn_meta (Dict[str, int]) – The dictionary saves information about group collation, including ‘num_denoising_queries’ and ‘num_denoising_groups’. It will be used for split outputs of denoising and matching parts and loss calculation.

Returns

A dictionary of loss components.

Return type

dict

loss_by_feat_single(cls_scores: Tensor, bbox_preds: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict]) → Tuple[Tensor]

Loss function for outputs from a single decoder layer of a single feature level.

Parameters

• cls_scores (Tensor) – Box score logits from a single decoder layer for all images, has shape (bs, num_queries, cls_out_channels).

• bbox_preds (Tensor) – Sigmoid outputs from a single decoder layer for all images, with normalized coordinate (cx, cy, w, h) and shape (bs, num_queries, 4).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

Returns

A tuple including loss_cls, loss_box and loss_iou.

Return type

Tuple[Tensor]

predict(hidden_states: Tensor, references: List[Tensor], memory_text: Tensor, text_token_mask: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[InstanceData]

Perform forward propagation and loss calculation of the detection head on the queries of the upstream network.

Parameters

• hidden_states (Tensor) – Hidden states output from each decoder layer, has shape (num_decoder_layers, num_queries, bs, dim).

• references (List[Tensor]) – List of the reference from the decoder. The first reference is the init_reference (initial) and the other num_decoder_layers(6) references are inter_references (intermediate). The init_reference has shape (bs, num_queries, 4) when as_two_stage of the detector is True, otherwise (bs, num_queries, 2). Each inter_reference has shape (bs, num_queries, 4) when with_box_refine of the detector is True, otherwise (bs, num_queries, 2). The coordinates are arranged as (cx, cy) when the last dimension is 2, and (cx, cy, w, h) when it is 4.

• memory_text (Tensor) – Memory text. It has shape (bs, len_text, text_embed_dims).

• text_token_mask (Tensor) – Text token mask. It has shape (bs, len_text).

• batch_data_samples (SampleList) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool, optional) – If True, return boxes in original image space. Defaults to True.

Returns

Detection results of each image

after the post process.

Return type

InstanceList

predict_by_feat(all_layers_cls_scores: Tensor, all_layers_bbox_preds: Tensor, batch_img_metas: List[Dict], batch_token_positive_maps: Optional[List[dict]] = None, rescale: bool = False) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Parameters

• all_layers_cls_scores (Tensor) – Classification scores of all decoder layers, has shape (num_decoder_layers, bs, num_queries, cls_out_channels).

• all_layers_bbox_preds (Tensor) – Regression outputs of all decoder layers. Each is a 4D-tensor with normalized coordinate format (cx, cy, w, h) and shape (num_decoder_layers, bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• batch_img_metas (List[Dict]) – _description_

• batch_token_positive_maps (list[dict], Optional) – Batch token positive map. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.dense_heads.GuidedAnchorHead(num_classes: int, in_channels: int, feat_channels: int = 256, approx_anchor_generator: Union[ConfigDict, dict] = {'octave_base_scale': 8, 'ratios': [0.5, 1.0, 2.0], 'scales_per_octave': 3, 'strides': [4, 8, 16, 32, 64], 'type': 'AnchorGenerator'}, square_anchor_generator: Union[ConfigDict, dict] = {'ratios': [1.0], 'scales': [8], 'strides': [4, 8, 16, 32, 64], 'type': 'AnchorGenerator'}, anchor_coder: Union[ConfigDict, dict] = {'target_means': [0.0, 0.0, 0.0, 0.0], 'target_stds': [1.0, 1.0, 1.0, 1.0], 'type': 'DeltaXYWHBBoxCoder'}, bbox_coder: Union[ConfigDict, dict] = {'target_means': [0.0, 0.0, 0.0, 0.0], 'target_stds': [1.0, 1.0, 1.0, 1.0], 'type': 'DeltaXYWHBBoxCoder'}, reg_decoded_bbox: bool = False, deform_groups: int = 4, loc_filter_thr: float = 0.01, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, loss_loc: Union[ConfigDict, dict] = {'alpha': 0.25, 'gamma': 2.0, 'loss_weight': 1.0, 'type': 'FocalLoss', 'use_sigmoid': True}, loss_shape: Union[ConfigDict, dict] = {'beta': 0.2, 'loss_weight': 1.0, 'type': 'BoundedIoULoss'}, loss_cls: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_bbox: Union[ConfigDict, dict] = {'beta': 1.0, 'loss_weight': 1.0, 'type': 'SmoothL1Loss'}, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'lbias_prob': 0.01, 'name': 'conv_loc', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'})

Guided-Anchor-based head (GA-RPN, GA-RetinaNet, etc.).

This GuidedAnchorHead will predict high-quality feature guided anchors and locations where anchors will be kept in inference. There are mainly 3 categories of bounding-boxes.

• Sampled 9 pairs for target assignment. (approxes)

• The square boxes where the predicted anchors are based on. (squares)

• Guided anchors.

Please refer to https://arxiv.org/abs/1901.03278 for more details.

Parameters

• num_classes (int) – Number of classes.

• in_channels (int) – Number of channels in the input feature map.

• feat_channels (int) – Number of hidden channels. Defaults to 256.

• approx_anchor_generator (ConfigDict or dict) – Config dict for approx generator

• square_anchor_generator (ConfigDict or dict) – Config dict for square generator

• anchor_coder (ConfigDict or dict) – Config dict for anchor coder

• bbox_coder (ConfigDict or dict) – Config dict for bbox coder

• reg_decoded_bbox (bool) – If true, the regression loss would be applied directly on decoded bounding boxes, converting both the predicted boxes and regression targets to absolute coordinates format. Defaults to False. It should be True when using IoULoss, GIoULoss, or DIoULoss in the bbox head.

• deform_groups – (int): Group number of DCN in FeatureAdaption module. Defaults to 4.

• loc_filter_thr (float) – Threshold to filter out unconcerned regions. Defaults to 0.01.

• loss_loc (ConfigDict or dict) – Config of location loss.

• loss_shape (ConfigDict or dict) – Config of anchor shape loss.

• loss_cls (ConfigDict or dict) – Config of classification loss.

• loss_bbox (ConfigDict or dict) – Config of bbox regression loss.

• init_cfg (ConfigDict or list[ConfigDict] or dict or list[dict], optional) – Initialization config dict.

forward(x: List[Tensor]) → Tuple[List[Tensor]]

Forward features from the upstream network.

forward_single(x: Tensor) → Tuple[Tensor]

Forward feature of a single scale level.

ga_loc_targets(batch_gt_instances: List[InstanceData], featmap_sizes: List[Tuple[int, int]]) → tuple

Compute location targets for guided anchoring.

Each feature map is divided into positive, negative and ignore regions. - positive regions: target 1, weight 1 - ignore regions: target 0, weight 0 - negative regions: target 0, weight 0.1

Parameters

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• featmap_sizes (list[tuple]) – Multi level sizes of each feature maps.

Returns

Returns a tuple containing location targets.

Return type

tuple

ga_shape_targets(approx_list: List[List[Tensor]], inside_flag_list: List[List[Tensor]], square_list: List[List[Tensor]], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs: bool = True) → tuple

Compute guided anchoring targets.

Parameters

• approx_list (list[list[Tensor]]) – Multi level approxs of each image.

• inside_flag_list (list[list[Tensor]]) – Multi level inside flags of each image.

• square_list (list[list[Tensor]]) – Multi level squares of each image.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• unmap_outputs (bool) – unmap outputs or not. Defaults to None.

Returns

Returns a tuple containing shape targets.

Return type

tuple

get_anchors(featmap_sizes: List[Tuple[int, int]], shape_preds: List[Tensor], loc_preds: List[Tensor], batch_img_metas: List[dict], use_loc_filter: bool = False, device: str = 'cuda') → tuple

Get squares according to feature map sizes and guided anchors.

Parameters

• featmap_sizes (list[tuple]) – Multi-level feature map sizes.

• shape_preds (list[tensor]) – Multi-level shape predictions.

• loc_preds (list[tensor]) – Multi-level location predictions.

• batch_img_metas (list[dict]) – Image meta info.

• use_loc_filter (bool) – Use loc filter or not. Defaults to False

• device (str) – device for returned tensors. Defaults to cuda.

Returns

square approxs of each image, guided anchors of each image, loc masks of each image.

Return type

tuple

get_sampled_approxs(featmap_sizes: List[Tuple[int, int]], batch_img_metas: List[dict], device: str = 'cuda') → tuple

Get sampled approxs and inside flags according to feature map sizes.

Parameters

• featmap_sizes (list[tuple]) – Multi-level feature map sizes.

• batch_img_metas (list[dict]) – Image meta info.

• device (str) – device for returned tensors

Returns

approxes of each image, inside flags of each image

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], shape_preds: List[Tensor], loc_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level has shape (N, num_anchors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• shape_preds (list[Tensor]) – shape predictions for each scale level with shape (N, 1, H, W).

• loc_preds (list[Tensor]) – location predictions for each scale level with shape (N, num_anchors * 2, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict

loss_loc_single(loc_pred: Tensor, loc_target: Tensor, loc_weight: Tensor, avg_factor: float) → Tensor

Compute location loss in single level.

loss_shape_single(shape_pred: Tensor, bbox_anchors: Tensor, bbox_gts: Tensor, anchor_weights: Tensor, avg_factor: int) → Tensor

Compute shape loss in single level.

predict_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], shape_preds: List[Tensor], loc_preds: List[Tensor], batch_img_metas: List[dict], cfg: Optional[Union[ConfigDict, dict]] = None, rescale: bool = False) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Parameters

• cls_scores (list[Tensor]) – Classification scores for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * 4, H, W).

• shape_preds (list[Tensor]) – shape predictions for each scale level with shape (N, 1, H, W).

• loc_preds (list[Tensor]) – location predictions for each scale level with shape (N, num_anchors * 2, H, W).

• batch_img_metas (list[dict], Optional) – Batch image meta info. Defaults to None.

• cfg (ConfigDict, optional) – Test / postprocessing configuration, if None, test_cfg would be used. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.dense_heads.LADHead(*args, topk: int = 9, score_voting: bool = True, covariance_type: str = 'diag', **kwargs)

Label Assignment Head from the paper: Improving Object Detection by Label Assignment Distillation

get_label_assignment(cls_scores: List[Tensor], bbox_preds: List[Tensor], iou_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → tuple

Get label assignment (from teacher).

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• iou_preds (list[Tensor]) – iou_preds for each scale level with shape (N, num_anchors * 1, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

Returns a tuple containing label assignment variables.

• labels (Tensor): Labels of all anchors, each with shape (num_anchors,).

• labels_weight (Tensor): Label weights of all anchor. each with shape (num_anchors,).

• bboxes_target (Tensor): BBox targets of all anchors. each with shape (num_anchors, 4).

• bboxes_weight (Tensor): BBox weights of all anchors. each with shape (num_anchors, 4).

• pos_inds_flatten (Tensor): Contains all index of positive sample in all anchor.

• pos_anchors (Tensor): Positive anchors.

• num_pos (int): Number of positive anchors.

Return type

tuple

loss(x: List[Tensor], label_assignment_results: tuple, batch_data_samples: List[DetDataSample]) → dict

Forward train with the available label assignment (student receives from teacher).

Parameters

• x (list[Tensor]) – Features from FPN.

• label_assignment_results (tuple) – As the outputs defined in the function self.get_label_assignment.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

(dict[str, Tensor]): A dictionary of loss components.

Return type

losses

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], iou_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, label_assignment_results: Optional[tuple] = None) → dict

Compute losses of the head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• iou_preds (list[Tensor]) – iou_preds for each scale level with shape (N, num_anchors * 1, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• label_assignment_results (tuple, optional) – As the outputs defined in the function self.get_ label_assignment.

Returns

A dictionary of loss gmm_assignment.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.LDHead(num_classes: int, in_channels: int, loss_ld: Union[ConfigDict, dict] = {'T': 10, 'loss_weight': 0.25, 'type': 'LocalizationDistillationLoss'}, **kwargs)

Localization distillation Head. (Short description)

It utilizes the learned bbox distributions to transfer the localization dark knowledge from teacher to student. Original paper: Localization Distillation for Object Detection.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• loss_ld (ConfigDict or dict) – Config of Localization Distillation Loss (LD), T is the temperature for distillation.

loss(x: List[Tensor], out_teacher: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → dict

Parameters

• x (list[Tensor]) – Features from FPN.

• out_teacher (tuple[Tensor]) – The output of teacher.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

The loss components and proposals of each image.

• losses (dict[str, Tensor]): A dictionary of loss components.

• proposal_list (list[Tensor]): Proposals of each image.

Return type

tuple[dict, list]

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], soft_targets: List[Tensor], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Compute losses of the head.

Parameters

• cls_scores (list[Tensor]) – Cls and quality scores for each scale level has shape (N, num_classes, H, W).

• bbox_preds (list[Tensor]) – Box distribution logits for each scale level with shape (N, 4*(n+1), H, W), n is max value of integral set.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• soft_targets (list[Tensor]) – Soft BBox regression targets.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_by_feat_single(anchors: Tensor, cls_score: Tensor, bbox_pred: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, stride: Tuple[int], soft_targets: Tensor, avg_factor: int)

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• anchors (Tensor) – Box reference for each scale level with shape (N, num_total_anchors, 4).

• cls_score (Tensor) – Cls and quality joint scores for each scale level has shape (N, num_classes, H, W).

• bbox_pred (Tensor) – Box distribution logits for each scale level with shape (N, 4*(n+1), H, W), n is max value of integral set.

• labels (Tensor) – Labels of each anchors with shape (N, num_total_anchors).

• label_weights (Tensor) – Label weights of each anchor with shape (N, num_total_anchors)

• bbox_targets (Tensor) – BBox regression targets of each anchor with shape (N, num_total_anchors, 4).

• stride (tuple) – Stride in this scale level.

• soft_targets (Tensor) – Soft BBox regression targets.

• avg_factor (int) – Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Returns

Loss components and weight targets.

Return type

dict[tuple, Tensor]

class mmdet.models.dense_heads.Mask2FormerHead(in_channels: List[int], feat_channels: int, out_channels: int, num_things_classes: int = 80, num_stuff_classes: int = 53, num_queries: int = 100, num_transformer_feat_level: int = 3, pixel_decoder: Union[ConfigDict, dict] = Ellipsis, enforce_decoder_input_project: bool = False, transformer_decoder: Union[ConfigDict, dict] = Ellipsis, positional_encoding: Union[ConfigDict, dict] = {'normalize': True, 'num_feats': 128}, loss_cls: Union[ConfigDict, dict] = {'class_weight': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.1], 'loss_weight': 2.0, 'reduction': 'mean', 'type': 'CrossEntropyLoss', 'use_sigmoid': False}, loss_mask: Union[ConfigDict, dict] = {'loss_weight': 5.0, 'reduction': 'mean', 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_dice: Union[ConfigDict, dict] = {'activate': True, 'eps': 1.0, 'loss_weight': 5.0, 'naive_dice': True, 'reduction': 'mean', 'type': 'DiceLoss', 'use_sigmoid': True}, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, **kwargs)

Implements the Mask2Former head.

See Masked-attention Mask Transformer for Universal Image Segmentation for details.

Parameters

• in_channels (list[int]) – Number of channels in the input feature map.

• feat_channels (int) – Number of channels for features.

• out_channels (int) – Number of channels for output.

• num_things_classes (int) – Number of things.

• num_stuff_classes (int) – Number of stuff.

• num_queries (int) – Number of query in Transformer decoder.

• pixel_decoder (ConfigDict or dict) – Config for pixel decoder. Defaults to None.

• enforce_decoder_input_project (bool, optional) – Whether to add a layer to change the embed_dim of transformer encoder in pixel decoder to the embed_dim of transformer decoder. Defaults to False.

• transformer_decoder (ConfigDict or dict) – Config for transformer decoder. Defaults to None.

• positional_encoding (ConfigDict or dict) – Config for transformer decoder position encoding. Defaults to dict(num_feats=128, normalize=True).

• loss_cls (ConfigDict or dict) – Config of the classification loss. Defaults to None.

• loss_mask (ConfigDict or dict) – Config of the mask loss. Defaults to None.

• loss_dice (ConfigDict or dict) – Config of the dice loss. Defaults to None.

• train_cfg (ConfigDict or dict, optional) – Training config of Mask2Former head.

• test_cfg (ConfigDict or dict, optional) – Testing config of Mask2Former head.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict. Defaults to None.

forward(x: List[Tensor], batch_data_samples: List[DetDataSample]) → Tuple[List[Tensor]]

Forward function.

Parameters

• x (list[Tensor]) – Multi scale Features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

A tuple contains two elements.

• cls_pred_list (list[Tensor)]: Classification logits for each decoder layer. Each is a 3D-tensor with shape (batch_size, num_queries, cls_out_channels). Note cls_out_channels should includes background.

• mask_pred_list (list[Tensor]): Mask logits for each decoder layer. Each with shape (batch_size, num_queries, h, w).

Return type

tuple[list[Tensor]]

init_weights() → None

Initialize the weights.

class mmdet.models.dense_heads.MaskFormerHead(in_channels: List[int], feat_channels: int, out_channels: int, num_things_classes: int = 80, num_stuff_classes: int = 53, num_queries: int = 100, pixel_decoder: Union[ConfigDict, dict] = Ellipsis, enforce_decoder_input_project: bool = False, transformer_decoder: Union[ConfigDict, dict] = Ellipsis, positional_encoding: Union[ConfigDict, dict] = {'normalize': True, 'num_feats': 128}, loss_cls: Union[ConfigDict, dict] = {'class_weight': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.1], 'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': False}, loss_mask: Union[ConfigDict, dict] = {'alpha': 0.25, 'gamma': 2.0, 'loss_weight': 20.0, 'type': 'FocalLoss', 'use_sigmoid': True}, loss_dice: Union[ConfigDict, dict] = {'activate': True, 'loss_weight': 1.0, 'naive_dice': True, 'type': 'DiceLoss', 'use_sigmoid': True}, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, **kwargs)

Implements the MaskFormer head.

See Per-Pixel Classification is Not All You Need for Semantic Segmentation for details.

Parameters

• in_channels (list[int]) – Number of channels in the input feature map.

• feat_channels (int) – Number of channels for feature.

• out_channels (int) – Number of channels for output.

• num_things_classes (int) – Number of things.

• num_stuff_classes (int) – Number of stuff.

• num_queries (int) – Number of query in Transformer.

• pixel_decoder (ConfigDict or dict) – Config for pixel decoder.

• enforce_decoder_input_project (bool) – Whether to add a layer to change the embed_dim of transformer encoder in pixel decoder to the embed_dim of transformer decoder. Defaults to False.

• transformer_decoder (ConfigDict or dict) – Config for transformer decoder.

• positional_encoding (ConfigDict or dict) – Config for transformer decoder position encoding.

• loss_cls (ConfigDict or dict) – Config of the classification loss. Defaults to CrossEntropyLoss.

• loss_mask (ConfigDict or dict) – Config of the mask loss. Defaults to FocalLoss.

• loss_dice (ConfigDict or dict) – Config of the dice loss. Defaults to DiceLoss.

• train_cfg (ConfigDict or dict, optional) – Training config of MaskFormer head.

• test_cfg (ConfigDict or dict, optional) – Testing config of MaskFormer head.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict. Defaults to None.

forward(x: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → Tuple[Tensor]

Forward function.

Parameters

• x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

a tuple contains two elements.

• all_cls_scores (Tensor): Classification scores for each scale level. Each is a 4D-tensor with shape (num_decoder, batch_size, num_queries, cls_out_channels). Note cls_out_channels should includes background.

• all_mask_preds (Tensor): Mask scores for each decoder layer. Each with shape (num_decoder, batch_size, num_queries, h, w).

Return type

tuple[Tensor]

get_targets(cls_scores_list: List[Tensor], mask_preds_list: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], return_sampling_results: bool = False) → Tuple[List[Union[Tensor, int]]]

Compute classification and mask targets for all images for a decoder layer.

Parameters

• cls_scores_list (list[Tensor]) – Mask score logits from a single decoder layer for all images. Each with shape (num_queries, cls_out_channels).

• mask_preds_list (list[Tensor]) – Mask logits from a single decoder layer for all images. Each with shape (num_queries, h, w).

• (list[obj (batch_gt_instances) – InstanceData]): each contains labels and masks.

• batch_img_metas (list[dict]) – List of image meta information.

• return_sampling_results (bool) – Whether to return the sampling results. Defaults to False.

Returns

a tuple containing the following targets.

• labels_list (list[Tensor]): Labels of all images. Each with shape (num_queries, ).

• label_weights_list (list[Tensor]): Label weights of all images. Each with shape (num_queries, ).

• mask_targets_list (list[Tensor]): Mask targets of all images. Each with shape (num_queries, h, w).

• mask_weights_list (list[Tensor]): Mask weights of all images. Each with shape (num_queries, ).

• avg_factor (int): Average factor that is used to average the loss. When using sampling method, avg_factor is

  usually the sum of positive and negative priors. When using MaskPseudoSampler, avg_factor is usually equal to the number of positive priors.

additional_returns: This function enables user-defined returns from

self._get_targets_single. These returns are currently refined to properties at each feature map (i.e. having HxW dimension). The results will be concatenated after the end.

Return type

tuple

init_weights() → None

Initialize the weights.

loss(x: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → Dict[str, Tensor]

Perform forward propagation and loss calculation of the panoptic head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Multi-level features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

a dictionary of loss components

Return type

dict[str, Tensor]

loss_by_feat(all_cls_scores: Tensor, all_mask_preds: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict]) → Dict[str, Tensor]

Loss function.

Parameters

• all_cls_scores (Tensor) – Classification scores for all decoder layers with shape (num_decoder, batch_size, num_queries, cls_out_channels). Note cls_out_channels should includes background.

• all_mask_preds (Tensor) – Mask scores for all decoder layers with shape (num_decoder, batch_size, num_queries, h, w).

• (list[obj (batch_gt_instances) – InstanceData]): each contains labels and masks.

• batch_img_metas (list[dict]) – List of image meta information.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

predict(x: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → Tuple[Tensor]

Test without augmentaton.

Parameters

• x (tuple[Tensor]) – Multi-level features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

A tuple contains two tensors.

• mask_cls_results (Tensor): Mask classification logits, shape (batch_size, num_queries, cls_out_channels).

  Note cls_out_channels should includes background.

• mask_pred_results (Tensor): Mask logits, shape (batch_size, num_queries, h, w).

Return type

tuple[Tensor]

preprocess_gt(batch_gt_instances: List[InstanceData], batch_gt_semantic_segs: List[Optional[PixelData]]) → List[InstanceData]

Preprocess the ground truth for all images.

Parameters

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes labels, each is ground truth labels of each bbox, with shape (num_gts, ) and masks, each is ground truth masks of each instances of a image, shape (num_gts, h, w).

• gt_semantic_seg (list[Optional[PixelData]]) – Ground truth of semantic segmentation, each with the shape (1, h, w). [0, num_thing_class - 1] means things, [num_thing_class, num_class-1] means stuff, 255 means VOID. It’s None when training instance segmentation.

Returns

InstanceData]: each contains the following keys

• labels (Tensor): Ground truth class indices for a image, with shape (n, ), n is the sum of number of stuff type and number of instance in a image.

• masks (Tensor): Ground truth mask for a image, with shape (n, h, w).

Return type

list[obj

class mmdet.models.dense_heads.NASFCOSHead(*args, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, **kwargs)

Anchor-free head used in NASFCOS.

It is quite similar with FCOS head, except for the searched structure of classification branch and bbox regression branch, where a structure of “dconv3x3, conv3x3, dconv3x3, conv1x1” is utilized instead.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• strides (Sequence[int] or Sequence[Tuple[int, int]]) – Strides of points in multiple feature levels. Defaults to (4, 8, 16, 32, 64).

• regress_ranges (Sequence[Tuple[int, int]]) – Regress range of multiple level points.

• center_sampling (bool) – If true, use center sampling. Defaults to False.

• center_sample_radius (float) – Radius of center sampling. Defaults to 1.5.

• norm_on_bbox (bool) – If true, normalize the regression targets with FPN strides. Defaults to False.

• centerness_on_reg (bool) – If true, position centerness on the regress branch. Please refer to https://github.com/tianzhi0549/FCOS/issues/89#issuecomment-516877042. Defaults to False.

• conv_bias (bool or str) – If specified as auto, it will be decided by the norm_cfg. Bias of conv will be set as True if norm_cfg is None, otherwise False. Defaults to “auto”.

• loss_cls (ConfigDict or dict) – Config of classification loss.

• loss_bbox (ConfigDict or dict) – Config of localization loss.

• loss_centerness (ConfigDict, or dict) – Config of centerness loss.

• norm_cfg (ConfigDict or dict) – dictionary to construct and config norm layer. Defaults to norm_cfg=dict(type='GN', num_groups=32, requires_grad=True).

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict.

class mmdet.models.dense_heads.PAAHead(*args, topk: int = 9, score_voting: bool = True, covariance_type: str = 'diag', **kwargs)

Head of PAAAssignment: Probabilistic Anchor Assignment with IoU Prediction for Object Detection.

Code is modified from the official github repo.

More details can be found in the paper .

Parameters

• topk (int) – Select topk samples with smallest loss in each level.

• score_voting (bool) – Whether to use score voting in post-process.

• covariance_type –

  String describing the type of covariance parameters to be used in sklearn.mixture.GaussianMixture. It must be one of:

  • ’full’: each component has its own general covariance matrix

  • ’tied’: all components share the same general covariance matrix

  • ’diag’: each component has its own diagonal covariance matrix

  • ’spherical’: each component has its own single variance

  Default: ‘diag’. From ‘full’ to ‘spherical’, the gmm fitting process is faster yet the performance could be influenced. For most cases, ‘diag’ should be a good choice.

get_pos_loss(anchors: List[Tensor], cls_score: Tensor, bbox_pred: Tensor, label: Tensor, label_weight: Tensor, bbox_target: dict, bbox_weight: Tensor, pos_inds: Tensor) → Tensor

Calculate loss of all potential positive samples obtained from first match process.

Parameters

• anchors (list[Tensor]) – Anchors of each scale.

• cls_score (Tensor) – Box scores of single image with shape (num_anchors, num_classes)

• bbox_pred (Tensor) – Box energies / deltas of single image with shape (num_anchors, 4)

• label (Tensor) – classification target of each anchor with shape (num_anchors,)

• label_weight (Tensor) – Classification loss weight of each anchor with shape (num_anchors).

• bbox_target (dict) – Regression target of each anchor with shape (num_anchors, 4).

• bbox_weight (Tensor) – Bbox weight of each anchor with shape (num_anchors, 4).

• pos_inds (Tensor) – Index of all positive samples got from first assign process.

Returns

Losses of all positive samples in single image.

Return type

Tensor

get_targets(anchor_list: List[List[Tensor]], valid_flag_list: List[List[Tensor]], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs: bool = True) → tuple

Get targets for PAA head.

This method is almost the same as AnchorHead.get_targets(). We direct return the results from _get_targets_single instead map it to levels by images_to_levels function.

Parameters

• anchor_list (list[list[Tensor]]) – Multi level anchors of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_anchors, 4).

• valid_flag_list (list[list[Tensor]]) – Multi level valid flags of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_anchors, )

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• unmap_outputs (bool) – Whether to map outputs back to the original set of anchors. Defaults to True.

Returns

Usually returns a tuple containing learning targets.

• labels (list[Tensor]): Labels of all anchors, each with

  shape (num_anchors,).

• label_weights (list[Tensor]): Label weights of all anchor.

  each with shape (num_anchors,).

• bbox_targets (list[Tensor]): BBox targets of all anchors.

  each with shape (num_anchors, 4).

• bbox_weights (list[Tensor]): BBox weights of all anchors.

  each with shape (num_anchors, 4).

• pos_inds (list[Tensor]): Contains all index of positive

  sample in all anchor.

• gt_inds (list[Tensor]): Contains all gt_index of positive

  sample in all anchor.

Return type

tuple

gmm_separation_scheme(gmm_assignment: Tensor, scores: Tensor, pos_inds_gmm: Tensor) → Tuple[Tensor, Tensor]

A general separation scheme for gmm model.

It separates a GMM distribution of candidate samples into three parts, 0 1 and uncertain areas, and you can implement other separation schemes by rewriting this function.

Parameters

• gmm_assignment (Tensor) – The prediction of GMM which is of shape (num_samples,). The 0/1 value indicates the distribution that each sample comes from.

• scores (Tensor) – The probability of sample coming from the fit GMM distribution. The tensor is of shape (num_samples,).

• pos_inds_gmm (Tensor) – All the indexes of samples which are used to fit GMM model. The tensor is of shape (num_samples,)

Returns

The indices of positive and ignored samples.

• pos_inds_temp (Tensor): Indices of positive samples.

• ignore_inds_temp (Tensor): Indices of ignore samples.

Return type

tuple[Tensor, Tensor]

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], iou_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• iou_preds (list[Tensor]) – iou_preds for each scale level with shape (N, num_anchors * 1, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss gmm_assignment.

Return type

dict[str, Tensor]

paa_reassign(pos_losses: Tensor, label: Tensor, label_weight: Tensor, bbox_weight: Tensor, pos_inds: Tensor, pos_gt_inds: Tensor, anchors: List[Tensor]) → tuple

Fit loss to GMM distribution and separate positive, ignore, negative samples again with GMM model.

Parameters

• pos_losses (Tensor) – Losses of all positive samples in single image.

• label (Tensor) – classification target of each anchor with shape (num_anchors,)

• label_weight (Tensor) – Classification loss weight of each anchor with shape (num_anchors).

• bbox_weight (Tensor) – Bbox weight of each anchor with shape (num_anchors, 4).

• pos_inds (Tensor) – Index of all positive samples got from first assign process.

• pos_gt_inds (Tensor) – Gt_index of all positive samples got from first assign process.

• anchors (list[Tensor]) – Anchors of each scale.

Returns

Usually returns a tuple containing learning targets.

• label (Tensor): classification target of each anchor after paa assign, with shape (num_anchors,)

• label_weight (Tensor): Classification loss weight of each anchor after paa assign, with shape (num_anchors).

• bbox_weight (Tensor): Bbox weight of each anchor with shape (num_anchors, 4).

• num_pos (int): The number of positive samples after paa assign.

Return type

tuple

predict_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], score_factors: Optional[List[Tensor]] = None, batch_img_metas: Optional[List[dict]] = None, cfg: Optional[Union[ConfigDict, dict]] = None, rescale: bool = False, with_nms: bool = True) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

This method is same as BaseDenseHead.get_results().

score_voting(det_bboxes: Tensor, det_labels: Tensor, mlvl_bboxes: Tensor, mlvl_nms_scores: Tensor, score_thr: float) → Tuple[Tensor, Tensor]

Implementation of score voting method works on each remaining boxes after NMS procedure.

Parameters

• det_bboxes (Tensor) – Remaining boxes after NMS procedure, with shape (k, 5), each dimension means (x1, y1, x2, y2, score).

• det_labels (Tensor) – The label of remaining boxes, with shape (k, 1),Labels are 0-based.

• mlvl_bboxes (Tensor) – All boxes before the NMS procedure, with shape (num_anchors,4).

• mlvl_nms_scores (Tensor) – The scores of all boxes which is used in the NMS procedure, with shape (num_anchors, num_class)

• score_thr (float) – The score threshold of bboxes.

Returns

Usually returns a tuple containing voting results.

• det_bboxes_voted (Tensor): Remaining boxes after

  score voting procedure, with shape (k, 5), each dimension means (x1, y1, x2, y2, score).

• det_labels_voted (Tensor): Label of remaining bboxes

  after voting, with shape (num_anchors,).

Return type

tuple

class mmdet.models.dense_heads.PISARetinaHead(num_classes, in_channels, stacked_convs=4, conv_cfg=None, norm_cfg=None, anchor_generator={'octave_base_scale': 4, 'ratios': [0.5, 1.0, 2.0], 'scales_per_octave': 3, 'strides': [8, 16, 32, 64, 128], 'type': 'AnchorGenerator'}, init_cfg={'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'retina_cls', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'}, **kwargs)

PISA Retinanet Head.

The head owns the same structure with Retinanet Head, but differs in two

aspects: 1. Importance-based Sample Reweighting Positive (ISR-P) is applied to

change the positive loss weights.

2. Classification-aware regression loss is adopted as a third loss.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Compute losses of the head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

Loss dict, comprise classification loss, regression loss and carl loss.

Return type

dict

class mmdet.models.dense_heads.PISASSDHead(num_classes: int = 80, in_channels: Sequence[int] = (512, 1024, 512, 256, 256, 256), stacked_convs: int = 0, feat_channels: int = 256, use_depthwise: bool = False, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, act_cfg: Optional[Union[ConfigDict, dict]] = None, anchor_generator: Union[ConfigDict, dict] = {'basesize_ratio_range': (0.1, 0.9), 'input_size': 300, 'ratios': ([2], [2, 3], [2, 3], [2, 3], [2], [2]), 'scale_major': False, 'strides': [8, 16, 32, 64, 100, 300], 'type': 'SSDAnchorGenerator'}, bbox_coder: Union[ConfigDict, dict] = {'clip_border': True, 'target_means': [0.0, 0.0, 0.0, 0.0], 'target_stds': [1.0, 1.0, 1.0, 1.0], 'type': 'DeltaXYWHBBoxCoder'}, reg_decoded_bbox: bool = False, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'bias': 0, 'distribution': 'uniform', 'layer': 'Conv2d', 'type': 'Xavier'})

Implementation of PISA SSD head

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (Sequence[int]) – Number of channels in the input feature map.

• stacked_convs (int) – Number of conv layers in cls and reg tower. Defaults to 0.

• feat_channels (int) – Number of hidden channels when stacked_convs > 0. Defaults to 256.

• use_depthwise (bool) – Whether to use DepthwiseSeparableConv. Defaults to False.

• conv_cfg (ConfigDict or dict, Optional) – Dictionary to construct and config conv layer. Defaults to None.

• norm_cfg (ConfigDict or dict, Optional) – Dictionary to construct and config norm layer. Defaults to None.

• act_cfg (ConfigDict or dict, Optional) – Dictionary to construct and config activation layer. Defaults to None.

• anchor_generator (ConfigDict or dict) – Config dict for anchor generator.

• bbox_coder (ConfigDict or dict) – Config of bounding box coder.

• reg_decoded_bbox (bool) – If true, the regression loss would be applied directly on decoded bounding boxes, converting both the predicted boxes and regression targets to absolute coordinates format. Defaults to False. It should be True when using IoULoss, GIoULoss, or DIoULoss in the bbox head.

• train_cfg (ConfigDict or dict, Optional) – Training config of anchor head.

• test_cfg (ConfigDict or dict, Optional) – Testing config of anchor head.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], Optional) – Initialization config dict.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Union[List[Tensor], Tensor]]

Compute losses of the head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components. the dict has components below:

• loss_cls (list[Tensor]): A list containing each feature map classification loss.

• loss_bbox (list[Tensor]): A list containing each feature map regression loss.

• loss_carl (Tensor): The loss of CARL.

Return type

dict[str, Union[List[Tensor], Tensor]]

class mmdet.models.dense_heads.RPNHead(in_channels: int, num_classes: int = 1, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'std': 0.01, 'type': 'Normal'}, num_convs: int = 1, **kwargs)

Implementation of RPN head.

Parameters

• in_channels (int) – Number of channels in the input feature map.

• num_classes (int) – Number of categories excluding the background category. Defaults to 1.

• init_cfg (ConfigDict or list[ConfigDict] or dict or list[dict]) – Initialization config dict.

• num_convs (int) – Number of convolution layers in the head. Defaults to 1.

forward_single(x: Tensor) → Tuple[Tensor, Tensor]

Forward feature of a single scale level.

Parameters

x (Tensor) – Features of a single scale level.

Returns

cls_score (Tensor): Cls scores for a single scale level the channels number is num_base_priors * num_classes. bbox_pred (Tensor): Box energies / deltas for a single scale level, the channels number is num_base_priors * 4.

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level, has shape (N, num_anchors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• (list[obj (batch_gt_instances_ignore) – InstanceData]): Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• (list[obj – InstanceData], Optional): Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.RTMDetHead(num_classes: int, in_channels: int, with_objectness: bool = True, act_cfg: Union[ConfigDict, dict] = {'type': 'ReLU'}, **kwargs)

Detection Head of RTMDet.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• with_objectness (bool) – Whether to add an objectness branch. Defaults to True.

• act_cfg (ConfigDict or dict) – Config dict for activation layer. Default: dict(type=’ReLU’)

forward(feats: Tuple[Tensor, ...]) → tuple

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually a tuple of classification scores and bbox prediction - cls_scores (list[Tensor]): Classification scores for all scale

levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

Return type

tuple

get_anchors(featmap_sizes: List[tuple], batch_img_metas: List[dict], device: Union[device, str] = 'cuda') → Tuple[List[List[Tensor]], List[List[Tensor]]]

Get anchors according to feature map sizes.

Parameters

• featmap_sizes (list[tuple]) – Multi-level feature map sizes.

• batch_img_metas (list[dict]) – Image meta info.

• device (torch.device or str) – Device for returned tensors. Defaults to cuda.

Returns

• anchor_list (list[list[Tensor]]): Anchors of each image.

• valid_flag_list (list[list[Tensor]]): Valid flags of each image.

Return type

tuple

get_targets(cls_scores: Tensor, bbox_preds: Tensor, anchor_list: List[List[Tensor]], valid_flag_list: List[List[Tensor]], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs=True)

Compute regression and classification targets for anchors in multiple images.

Parameters

• cls_scores (Tensor) – Classification predictions of images, a 3D-Tensor with shape [num_imgs, num_priors, num_classes].

• bbox_preds (Tensor) – Decoded bboxes predictions of one image, a 3D-Tensor with shape [num_imgs, num_priors, 4] in [tl_x, tl_y, br_x, br_y] format.

• anchor_list (list[list[Tensor]]) – Multi level anchors of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_anchors, 4).

• valid_flag_list (list[list[Tensor]]) – Multi level valid flags of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_anchors, )

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• unmap_outputs (bool) – Whether to map outputs back to the original set of anchors. Defaults to True.

Returns

a tuple containing learning targets.

• anchors_list (list[list[Tensor]]): Anchors of each level.

• labels_list (list[Tensor]): Labels of each level.

• label_weights_list (list[Tensor]): Label weights of each level.

• bbox_targets_list (list[Tensor]): BBox targets of each level.

• assign_metrics_list (list[Tensor]): alignment metrics of each level.

Return type

tuple

init_weights() → None

Initialize weights of the head.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None)

Compute losses of the head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Decoded box for each scale level with shape (N, num_anchors * 4, H, W) in [tl_x, tl_y, br_x, br_y] format.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_by_feat_single(cls_score: Tensor, bbox_pred: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, assign_metrics: Tensor, stride: List[int])

Compute loss of a single scale level.

Parameters

• cls_score (Tensor) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W).

• bbox_pred (Tensor) – Decoded bboxes for each scale level with shape (N, num_anchors * 4, H, W).

• labels (Tensor) – Labels of each anchors with shape (N, num_total_anchors).

• label_weights (Tensor) – Label weights of each anchor with shape (N, num_total_anchors).

• bbox_targets (Tensor) – BBox regression targets of each anchor with shape (N, num_total_anchors, 4).

• assign_metrics (Tensor) – Assign metrics with shape (N, num_total_anchors).

• stride (List[int]) – Downsample stride of the feature map.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.RTMDetInsHead(*args, num_prototypes: int = 8, dyconv_channels: int = 8, num_dyconvs: int = 3, mask_loss_stride: int = 4, loss_mask={'eps': 5e-06, 'loss_weight': 2.0, 'reduction': 'mean', 'type': 'DiceLoss'}, **kwargs)

Detection Head of RTMDet-Ins.

Parameters

• num_prototypes (int) – Number of mask prototype features extracted from the mask head. Defaults to 8.

• dyconv_channels (int) – Channel of the dynamic conv layers. Defaults to 8.

• num_dyconvs (int) – Number of the dynamic convolution layers. Defaults to 3.

• mask_loss_stride (int) – Down sample stride of the masks for loss computation. Defaults to 4.

• loss_mask (ConfigDict or dict) – Config dict for mask loss.

forward(feats: Tuple[Tensor, ...]) → tuple

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually a tuple of classification scores and bbox prediction - cls_scores (list[Tensor]): Classification scores for all scale

levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

• kernel_preds (list[Tensor]): Dynamic conv kernels for all scale levels, each is a 4D-tensor, the channels number is num_gen_params.

• mask_feat (Tensor): Output feature of the mask head. Each is a 4D-tensor, the channels number is num_prototypes.

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], kernel_preds: List[Tensor], mask_feat: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None)

Compute losses of the head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Decoded box for each scale level with shape (N, num_anchors * 4, H, W) in [tl_x, tl_y, br_x, br_y] format.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_mask_by_feat(mask_feats: Tensor, flatten_kernels: Tensor, sampling_results_list: list, batch_gt_instances: List[InstanceData]) → Tensor

Compute instance segmentation loss.

Parameters

• mask_feats (list[Tensor]) – Mask prototype features extracted from the mask head. Has shape (N, num_prototypes, H, W)

• flatten_kernels (list[Tensor]) – Kernels of the dynamic conv layers. Has shape (N, num_instances, num_params)

• sampling_results_list (list[SamplingResults]) – assignment results.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

Returns

The mask loss tensor.

Return type

Tensor

parse_dynamic_params(flatten_kernels: Tensor) → tuple

Split kernel head prediction to conv weight and bias.

predict_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], kernel_preds: List[Tensor], mask_feat: Tensor, score_factors: Optional[List[Tensor]] = None, batch_img_metas: Optional[List[dict]] = None, cfg: Optional[Union[ConfigDict, dict]] = None, rescale: bool = False, with_nms: bool = True) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Note: When score_factors is not None, the cls_scores are usually multiplied by it then obtain the real score used in NMS, such as CenterNess in FCOS, IoU branch in ATSS.

Parameters

• cls_scores (list[Tensor]) – Classification scores for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * 4, H, W).

• kernel_preds (list[Tensor]) – Kernel predictions of dynamic convs for all scale levels, each is a 4D-tensor, has shape (batch_size, num_params, H, W).

• mask_feat (Tensor) – Mask prototype features extracted from the mask head, has shape (batch_size, num_prototypes, H, W).

• score_factors (list[Tensor], optional) – Score factor for all scale level, each is a 4D-tensor, has shape (batch_size, num_priors * 1, H, W). Defaults to None.

• batch_img_metas (list[dict], Optional) – Batch image meta info. Defaults to None.

• cfg (ConfigDict, optional) – Test / postprocessing configuration, if None, test_cfg would be used. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

• with_nms (bool) – If True, do nms before return boxes. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, h, w).

Return type

list[InstanceData]

class mmdet.models.dense_heads.RTMDetInsSepBNHead(num_classes: int, in_channels: int, share_conv: bool = True, with_objectness: bool = False, norm_cfg: Union[ConfigDict, dict] = {'requires_grad': True, 'type': 'BN'}, act_cfg: Union[ConfigDict, dict] = {'inplace': True, 'type': 'SiLU'}, pred_kernel_size: int = 1, **kwargs)

Detection Head of RTMDet-Ins with sep-bn layers.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• share_conv (bool) – Whether to share conv layers between stages. Defaults to True.

• norm_cfg (ConfigDict or dict)) – Config dict for normalization layer. Defaults to dict(type=’BN’).

• act_cfg (ConfigDict or dict)) – Config dict for activation layer. Defaults to dict(type=’SiLU’, inplace=True).

• pred_kernel_size (int) – Kernel size of prediction layer. Defaults to 1.

forward(feats: Tuple[Tensor, ...]) → tuple

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually a tuple of classification scores and bbox prediction - cls_scores (list[Tensor]): Classification scores for all scale

levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

• kernel_preds (list[Tensor]): Dynamic conv kernels for all scale levels, each is a 4D-tensor, the channels number is num_gen_params.

• mask_feat (Tensor): Output feature of the mask head. Each is a 4D-tensor, the channels number is num_prototypes.

Return type

tuple

init_weights() → None

Initialize weights of the head.

class mmdet.models.dense_heads.RTMDetSepBNHead(num_classes: int, in_channels: int, share_conv: bool = True, use_depthwise: bool = False, norm_cfg: Union[ConfigDict, dict] = {'eps': 0.001, 'momentum': 0.03, 'type': 'BN'}, act_cfg: Union[ConfigDict, dict] = {'type': 'SiLU'}, pred_kernel_size: int = 1, exp_on_reg=False, **kwargs)

RTMDetHead with separated BN layers and shared conv layers.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• share_conv (bool) – Whether to share conv layers between stages. Defaults to True.

• use_depthwise (bool) – Whether to use depthwise separable convolution in head. Defaults to False.

• norm_cfg (ConfigDict or dict)) – Config dict for normalization layer. Defaults to dict(type=’BN’, momentum=0.03, eps=0.001).

• act_cfg (ConfigDict or dict)) – Config dict for activation layer. Defaults to dict(type=’SiLU’).

• pred_kernel_size (int) – Kernel size of prediction layer. Defaults to 1.

forward(feats: Tuple[Tensor, ...]) → tuple

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually a tuple of classification scores and bbox prediction

• cls_scores (tuple[Tensor]): Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_anchors * num_classes.

• bbox_preds (tuple[Tensor]): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_anchors * 4.

Return type

tuple

init_weights() → None

Initialize weights of the head.

class mmdet.models.dense_heads.RepPointsHead(num_classes: int, in_channels: int, point_feat_channels: int = 256, num_points: int = 9, gradient_mul: float = 0.1, point_strides: Sequence[int] = [8, 16, 32, 64, 128], point_base_scale: int = 4, loss_cls: Union[ConfigDict, dict] = {'alpha': 0.25, 'gamma': 2.0, 'loss_weight': 1.0, 'type': 'FocalLoss', 'use_sigmoid': True}, loss_bbox_init: Union[ConfigDict, dict] = {'beta': 0.1111111111111111, 'loss_weight': 0.5, 'type': 'SmoothL1Loss'}, loss_bbox_refine: Union[ConfigDict, dict] = {'beta': 0.1111111111111111, 'loss_weight': 1.0, 'type': 'SmoothL1Loss'}, use_grid_points: bool = False, center_init: bool = True, transform_method: str = 'moment', moment_mul: float = 0.01, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'reppoints_cls_out', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'}, **kwargs)

RepPoint head.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• point_feat_channels (int) – Number of channels of points features.

• num_points (int) – Number of points.

• gradient_mul (float) – The multiplier to gradients from points refinement and recognition.

• point_strides (Sequence[int]) – points strides.

• point_base_scale (int) – bbox scale for assigning labels.

• loss_cls (ConfigDict or dict) – Config of classification loss.

• loss_bbox_init (ConfigDict or dict) – Config of initial points loss.

• loss_bbox_refine (ConfigDict or dict) – Config of points loss in refinement.

• use_grid_points (bool) – If we use bounding box representation, the

• box. (reppoints is represented as grid points on the bounding) –

• center_init (bool) – Whether to use center point assignment.

• transform_method (str) – The methods to transform RepPoints to bbox.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict]) – Initialization config dict.

centers_to_bboxes(point_list: List[Tensor]) → List[Tensor]

Get bboxes according to center points.

Only used in MaxIoUAssigner.

forward(feats: Tuple[Tensor]) → Tuple[Tensor]

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually contain classification scores and bbox predictions.

• cls_scores (list[Tensor]): Box scores for each scale level, each is a 4D-tensor, the channel number is num_points * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

Return type

tuple

forward_single(x: Tensor) → Tuple[Tensor]

Forward feature map of a single FPN level.

gen_grid_from_reg(reg: Tensor, previous_boxes: Tensor) → Tuple[Tensor]

Base on the previous bboxes and regression values, we compute the regressed bboxes and generate the grids on the bboxes.

Parameters

• reg (Tensor) – the regression value to previous bboxes.

• previous_boxes (Tensor) – previous bboxes.

Returns

generate grids on the regressed bboxes.

Return type

Tuple[Tensor]

get_points(featmap_sizes: List[Tuple[int]], batch_img_metas: List[dict], device: str) → tuple

Get points according to feature map sizes.

Parameters

• featmap_sizes (list[tuple]) – Multi-level feature map sizes.

• batch_img_metas (list[dict]) – Image meta info.

Returns

points of each image, valid flags of each image

Return type

tuple

get_targets(proposals_list: List[Tensor], valid_flag_list: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, stage: str = 'init', unmap_outputs: bool = True, return_sampling_results: bool = False) → tuple

Compute corresponding GT box and classification targets for proposals.

Parameters

• proposals_list (list[Tensor]) – Multi level points/bboxes of each image.

• valid_flag_list (list[Tensor]) – Multi level valid flags of each image.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• stage (str) – ‘init’ or ‘refine’. Generate target for init stage or refine stage.

• unmap_outputs (bool) – Whether to map outputs back to the original set of anchors.

• return_sampling_results (bool) – Whether to return the sampling results. Defaults to False.

Returns

• labels_list (list[Tensor]): Labels of each level.

• label_weights_list (list[Tensor]): Label weights of each level.

• bbox_gt_list (list[Tensor]): Ground truth bbox of each level.

• proposals_list (list[Tensor]): Proposals(points/bboxes) of each level.

• proposal_weights_list (list[Tensor]): Proposal weights of each level.

• avg_factor (int): Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], pts_preds_init: List[Tensor], pts_preds_refine: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level, each is a 4D-tensor, of shape (batch_size, num_classes, h, w).

• pts_preds_init (list[Tensor]) – Points for each scale level, each is a 3D-tensor, of shape (batch_size, h_i * w_i, num_points * 2).

• pts_preds_refine (list[Tensor]) – Points refined for each scale level, each is a 3D-tensor, of shape (batch_size, h_i * w_i, num_points * 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_by_feat_single(cls_score: Tensor, pts_pred_init: Tensor, pts_pred_refine: Tensor, labels: Tensor, label_weights, bbox_gt_init: Tensor, bbox_weights_init: Tensor, bbox_gt_refine: Tensor, bbox_weights_refine: Tensor, stride: int, avg_factor_init: int, avg_factor_refine: int) → Tuple[Tensor]

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• cls_score (Tensor) – Box scores for each scale level Has shape (N, num_classes, h_i, w_i).

• pts_pred_init (Tensor) – Points of shape (batch_size, h_i * w_i, num_points * 2).

• pts_pred_refine (Tensor) – Points refined of shape (batch_size, h_i * w_i, num_points * 2).

• labels (Tensor) – Ground truth class indices with shape (batch_size, h_i * w_i).

• label_weights (Tensor) – Label weights of shape (batch_size, h_i * w_i).

• bbox_gt_init (Tensor) – BBox regression targets in the init stage of shape (batch_size, h_i * w_i, 4).

• bbox_weights_init (Tensor) – BBox regression loss weights in the init stage of shape (batch_size, h_i * w_i, 4).

• bbox_gt_refine (Tensor) – BBox regression targets in the refine stage of shape (batch_size, h_i * w_i, 4).

• bbox_weights_refine (Tensor) – BBox regression loss weights in the refine stage of shape (batch_size, h_i * w_i, 4).

• stride (int) – Point stride.

• avg_factor_init (int) – Average factor that is used to average the loss in the init stage.

• avg_factor_refine (int) – Average factor that is used to average the loss in the refine stage.

Returns

loss components.

Return type

Tuple[Tensor]

offset_to_pts(center_list: List[Tensor], pred_list: List[Tensor]) → List[Tensor]

Change from point offset to point coordinate.

points2bbox(pts: Tensor, y_first: bool = True) → Tensor

Converting the points set into bounding box.

Parameters

• pts (Tensor) – the input points sets (fields), each points set (fields) is represented as 2n scalar.

• y_first (bool) – if y_first=True, the point set is represented as [y1, x1, y2, x2 … yn, xn], otherwise the point set is represented as [x1, y1, x2, y2 … xn, yn]. Defaults to True.

Returns

each points set is converting to a bbox [x1, y1, x2, y2].

Return type

Tensor

class mmdet.models.dense_heads.RetinaHead(num_classes, in_channels, stacked_convs=4, conv_cfg=None, norm_cfg=None, anchor_generator={'octave_base_scale': 4, 'ratios': [0.5, 1.0, 2.0], 'scales_per_octave': 3, 'strides': [8, 16, 32, 64, 128], 'type': 'AnchorGenerator'}, init_cfg={'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'retina_cls', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'}, **kwargs)

An anchor-based head used in RetinaNet.

The head contains two subnetworks. The first classifies anchor boxes and the second regresses deltas for the anchors.

Example

>>> import torch
>>> self = RetinaHead(11, 7)
>>> x = torch.rand(1, 7, 32, 32)
>>> cls_score, bbox_pred = self.forward_single(x)
>>> # Each anchor predicts a score for each class except background
>>> cls_per_anchor = cls_score.shape[1] / self.num_anchors
>>> box_per_anchor = bbox_pred.shape[1] / self.num_anchors
>>> assert cls_per_anchor == (self.num_classes)
>>> assert box_per_anchor == 4

forward_single(x)

Forward feature of a single scale level.

Parameters

x (Tensor) – Features of a single scale level.

Returns

cls_score (Tensor): Cls scores for a single scale level

the channels number is num_anchors * num_classes.

bbox_pred (Tensor): Box energies / deltas for a single scale

level, the channels number is num_anchors * 4.

Return type

tuple

class mmdet.models.dense_heads.RetinaSepBNHead(num_classes: int, num_ins: int, in_channels: int, stacked_convs: int = 4, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, **kwargs)

“RetinaHead with separate BN.

In RetinaHead, conv/norm layers are shared across different FPN levels, while in RetinaSepBNHead, conv layers are shared across different FPN levels, but BN layers are separated.

forward(feats: Tuple[Tensor]) → tuple

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually a tuple of classification scores and bbox prediction

• cls_scores (list[Tensor]): Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_anchors * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_anchors * 4.

Return type

tuple

init_weights() → None

Initialize weights of the head.

class mmdet.models.dense_heads.SABLRetinaHead(num_classes: int, in_channels: int, stacked_convs: int = 4, feat_channels: int = 256, approx_anchor_generator: Union[ConfigDict, dict] = {'octave_base_scale': 4, 'ratios': [0.5, 1.0, 2.0], 'scales_per_octave': 3, 'strides': [8, 16, 32, 64, 128], 'type': 'AnchorGenerator'}, square_anchor_generator: Union[ConfigDict, dict] = {'ratios': [1.0], 'scales': [4], 'strides': [8, 16, 32, 64, 128], 'type': 'AnchorGenerator'}, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, bbox_coder: Union[ConfigDict, dict] = {'num_buckets': 14, 'scale_factor': 3.0, 'type': 'BucketingBBoxCoder'}, reg_decoded_bbox: bool = False, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, loss_cls: Union[ConfigDict, dict] = {'alpha': 0.25, 'gamma': 2.0, 'loss_weight': 1.0, 'type': 'FocalLoss', 'use_sigmoid': True}, loss_bbox_cls: Union[ConfigDict, dict] = {'loss_weight': 1.5, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_bbox_reg: Union[ConfigDict, dict] = {'beta': 0.1111111111111111, 'loss_weight': 1.5, 'type': 'SmoothL1Loss'}, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'retina_cls', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'})

Side-Aware Boundary Localization (SABL) for RetinaNet.

The anchor generation, assigning and sampling in SABLRetinaHead are the same as GuidedAnchorHead for guided anchoring.

Please refer to https://arxiv.org/abs/1912.04260 for more details.

Parameters

• num_classes (int) – Number of classes.

• in_channels (int) – Number of channels in the input feature map.

• stacked_convs (int) – Number of Convs for classification and regression branches. Defaults to 4.

• feat_channels (int) – Number of hidden channels. Defaults to 256.

• approx_anchor_generator (ConfigType or dict) – Config dict for approx generator.

• square_anchor_generator (ConfigDict or dict) – Config dict for square generator.

• conv_cfg (ConfigDict or dict, optional) – Config dict for ConvModule. Defaults to None.

• norm_cfg (ConfigDict or dict, optional) – Config dict for Norm Layer. Defaults to None.

• bbox_coder (ConfigDict or dict) – Config dict for bbox coder.

• reg_decoded_bbox (bool) – If true, the regression loss would be applied directly on decoded bounding boxes, converting both the predicted boxes and regression targets to absolute coordinates format. Default False. It should be True when using IoULoss, GIoULoss, or DIoULoss in the bbox head.

• train_cfg (ConfigDict or dict, optional) – Training config of SABLRetinaHead.

• test_cfg (ConfigDict or dict, optional) – Testing config of SABLRetinaHead.

• loss_cls (ConfigDict or dict) – Config of classification loss.

• loss_bbox_cls (ConfigDict or dict) – Config of classification loss for bbox branch.

• loss_bbox_reg (ConfigDict or dict) – Config of regression loss for bbox branch.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict.

forward(feats: List[Tensor]) → Tuple[List[Tensor]]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

get_anchors(featmap_sizes: List[tuple], img_metas: List[dict], device: Union[device, str] = 'cuda') → Tuple[List[List[Tensor]], List[List[Tensor]]]

Get squares according to feature map sizes and guided anchors.

Parameters

• featmap_sizes (list[tuple]) – Multi-level feature map sizes.

• img_metas (list[dict]) – Image meta info.

• device (torch.device | str) – device for returned tensors

Returns

square approxs of each image

Return type

tuple

get_targets(approx_list: List[List[Tensor]], inside_flag_list: List[List[Tensor]], square_list: List[List[Tensor]], batch_gt_instances: List[InstanceData], batch_img_metas, batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs=True) → tuple

Compute bucketing targets.

Parameters

• approx_list (list[list[Tensor]]) – Multi level approxs of each image.

• inside_flag_list (list[list[Tensor]]) – Multi level inside flags of each image.

• square_list (list[list[Tensor]]) – Multi level squares of each image.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• unmap_outputs (bool) – Whether to map outputs back to the original set of anchors. Defaults to True.

Returns

Returns a tuple containing learning targets.

• labels_list (list[Tensor]): Labels of each level.

• label_weights_list (list[Tensor]): Label weights of each level.

• bbox_cls_targets_list (list[Tensor]): BBox cls targets of each level.

• bbox_cls_weights_list (list[Tensor]): BBox cls weights of each level.

• bbox_reg_targets_list (list[Tensor]): BBox reg targets of each level.

• bbox_reg_weights_list (list[Tensor]): BBox reg weights of each level.

• num_total_pos (int): Number of positive samples in all images.

• num_total_neg (int): Number of negative samples in all images.

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level has shape (N, num_anchors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict

loss_by_feat_single(cls_score: Tensor, bbox_pred: Tensor, labels: Tensor, label_weights: Tensor, bbox_cls_targets: Tensor, bbox_cls_weights: Tensor, bbox_reg_targets: Tensor, bbox_reg_weights: Tensor, avg_factor: float) → Tuple[Tensor]

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• cls_score (Tensor) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W).

• bbox_pred (Tensor) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• labels (Tensor) – Labels in a single image.

• label_weights (Tensor) – Label weights in a single level.

• bbox_cls_targets (Tensor) – BBox cls targets in a single level.

• bbox_cls_weights (Tensor) – BBox cls weights in a single level.

• bbox_reg_targets (Tensor) – BBox reg targets in a single level.

• bbox_reg_weights (Tensor) – BBox reg weights in a single level.

• avg_factor (int) – Average factor that is used to average the loss.

Returns

loss components.

Return type

tuple

predict_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_img_metas: List[dict], cfg: Optional[ConfigDict] = None, rescale: bool = False, with_nms: bool = True) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Note: When score_factors is not None, the cls_scores are usually multiplied by it then obtain the real score used in NMS, such as CenterNess in FCOS, IoU branch in ATSS.

Parameters

• cls_scores (list[Tensor]) – Classification scores for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * 4, H, W).

• batch_img_metas (list[dict], Optional) – Batch image meta info.

• cfg (ConfigDict, optional) – Test / postprocessing configuration, if None, test_cfg would be used. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

• with_nms (bool) – If True, do nms before return boxes. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.dense_heads.SOLOHead(num_classes: int, in_channels: int, feat_channels: int = 256, stacked_convs: int = 4, strides: tuple = (4, 8, 16, 32, 64), scale_ranges: tuple = ((8, 32), (16, 64), (32, 128), (64, 256), (128, 512)), pos_scale: float = 0.2, num_grids: list = [40, 36, 24, 16, 12], cls_down_index: int = 0, loss_mask: Union[ConfigDict, dict] = {'loss_weight': 3.0, 'type': 'DiceLoss', 'use_sigmoid': True}, loss_cls: Union[ConfigDict, dict] = {'alpha': 0.25, 'gamma': 2.0, 'loss_weight': 1.0, 'type': 'FocalLoss', 'use_sigmoid': True}, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 32, 'requires_grad': True, 'type': 'GN'}, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = [{'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}, {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01, 'override': {'name': 'conv_mask_list'}}, {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01, 'override': {'name': 'conv_cls'}}])

SOLO mask head used in `SOLO: Segmenting Objects by Locations.

<https://arxiv.org/abs/1912.04488>`_

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• feat_channels (int) – Number of hidden channels. Used in child classes. Defaults to 256.

• stacked_convs (int) – Number of stacking convs of the head. Defaults to 4.

• strides (tuple) – Downsample factor of each feature map.

• scale_ranges (tuple[tuple[int, int]]) – Area range of multiple level masks, in the format [(min1, max1), (min2, max2), …]. A range of (16, 64) means the area range between (16, 64).

• pos_scale (float) – Constant scale factor to control the center region.

• num_grids (list[int]) – Divided image into a uniform grids, each feature map has a different grid value. The number of output channels is grid ** 2. Defaults to [40, 36, 24, 16, 12].

• cls_down_index (int) – The index of downsample operation in classification branch. Defaults to 0.

• loss_mask (dict) – Config of mask loss.

• loss_cls (dict) – Config of classification loss.

• norm_cfg (dict) – Dictionary to construct and config norm layer. Defaults to norm_cfg=dict(type=’GN’, num_groups=32, requires_grad=True).

• train_cfg (dict) – Training config of head.

• test_cfg (dict) – Testing config of head.

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(x: Tuple[Tensor]) → tuple

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of classification scores and mask prediction.

• mlvl_mask_preds (list[Tensor]): Multi-level mask prediction. Each element in the list has shape (batch_size, num_grids**2 ,h ,w).

• mlvl_cls_preds (list[Tensor]): Multi-level scores. Each element in the list has shape (batch_size, num_classes, num_grids ,num_grids).

Return type

tuple

loss_by_feat(mlvl_mask_preds: List[Tensor], mlvl_cls_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], **kwargs) → dict

Calculate the loss based on the features extracted by the mask head.

Parameters

• mlvl_mask_preds (list[Tensor]) – Multi-level mask prediction. Each element in the list has shape (batch_size, num_grids**2 ,h ,w).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, masks, and labels attributes.

• batch_img_metas (list[dict]) – Meta information of multiple images.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

predict_by_feat(mlvl_mask_preds: List[Tensor], mlvl_cls_scores: List[Tensor], batch_img_metas: List[dict], **kwargs) → List[InstanceData]

Transform a batch of output features extracted from the head into mask results.

Parameters

• mlvl_mask_preds (list[Tensor]) – Multi-level mask prediction. Each element in the list has shape (batch_size, num_grids**2 ,h ,w).

• mlvl_cls_scores (list[Tensor]) – Multi-level scores. Each element in the list has shape (batch_size, num_classes, num_grids ,num_grids).

• batch_img_metas (list[dict]) – Meta information of all images.

Returns

Processed results of multiple images.Each InstanceData usually contains following keys.

• scores (Tensor): Classification scores, has shape (num_instance,).

• labels (Tensor): Has shape (num_instances,).

• masks (Tensor): Processed mask results, has shape (num_instances, h, w).

Return type

list[InstanceData]

resize_feats(x: Tuple[Tensor]) → List[Tensor]

Downsample the first feat and upsample last feat in feats.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Features after resizing, each is a 4D-tensor.

Return type

list[Tensor]

class mmdet.models.dense_heads.SOLOV2Head(*args, mask_feature_head: Union[ConfigDict, dict], dynamic_conv_size: int = 1, dcn_cfg: Optional[Union[ConfigDict, dict]] = None, dcn_apply_to_all_conv: bool = True, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = [{'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}, {'type': 'Normal', 'std': 0.01, 'bias_prob': 0.01, 'override': {'name': 'conv_cls'}}], **kwargs)

SOLOv2 mask head used in SOLOv2: Dynamic and Fast Instance Segmentation.

Parameters

• mask_feature_head (dict) – Config of SOLOv2MaskFeatHead.

• dynamic_conv_size (int) – Dynamic Conv kernel size. Defaults to 1.

• dcn_cfg (dict) – Dcn conv configurations in kernel_convs and cls_conv. Defaults to None.

• dcn_apply_to_all_conv (bool) – Whether to use dcn in every layer of kernel_convs and cls_convs, or only the last layer. It shall be set True for the normal version of SOLOv2 and False for the light-weight version. Defaults to True.

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(x)

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of classification scores, mask prediction, and mask features.

• mlvl_kernel_preds (list[Tensor]): Multi-level dynamic kernel prediction. The kernel is used to generate instance segmentation masks by dynamic convolution. Each element in the list has shape (batch_size, kernel_out_channels, num_grids, num_grids).

• mlvl_cls_preds (list[Tensor]): Multi-level scores. Each element in the list has shape (batch_size, num_classes, num_grids, num_grids).

• mask_feats (Tensor): Unified mask feature map used to generate instance segmentation masks by dynamic convolution. Has shape (batch_size, mask_out_channels, h, w).

Return type

tuple

loss_by_feat(mlvl_kernel_preds: List[Tensor], mlvl_cls_preds: List[Tensor], mask_feats: Tensor, batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], **kwargs) → dict

Calculate the loss based on the features extracted by the mask head.

Parameters

• mlvl_kernel_preds (list[Tensor]) – Multi-level dynamic kernel prediction. The kernel is used to generate instance segmentation masks by dynamic convolution. Each element in the list has shape (batch_size, kernel_out_channels, num_grids, num_grids).

• mlvl_cls_preds (list[Tensor]) – Multi-level scores. Each element in the list has shape (batch_size, num_classes, num_grids, num_grids).

• mask_feats (Tensor) – Unified mask feature map used to generate instance segmentation masks by dynamic convolution. Has shape (batch_size, mask_out_channels, h, w).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, masks, and labels attributes.

• batch_img_metas (list[dict]) – Meta information of multiple images.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

predict_by_feat(mlvl_kernel_preds: List[Tensor], mlvl_cls_scores: List[Tensor], mask_feats: Tensor, batch_img_metas: List[dict], **kwargs) → List[InstanceData]

Transform a batch of output features extracted from the head into mask results.

Parameters

• mlvl_kernel_preds (list[Tensor]) – Multi-level dynamic kernel prediction. The kernel is used to generate instance segmentation masks by dynamic convolution. Each element in the list has shape (batch_size, kernel_out_channels, num_grids, num_grids).

• mlvl_cls_scores (list[Tensor]) – Multi-level scores. Each element in the list has shape (batch_size, num_classes, num_grids, num_grids).

• mask_feats (Tensor) – Unified mask feature map used to generate instance segmentation masks by dynamic convolution. Has shape (batch_size, mask_out_channels, h, w).

• batch_img_metas (list[dict]) – Meta information of all images.

Returns

Processed results of multiple images.Each InstanceData usually contains following keys.

• scores (Tensor): Classification scores, has shape (num_instance,).

• labels (Tensor): Has shape (num_instances,).

• masks (Tensor): Processed mask results, has shape (num_instances, h, w).

Return type

list[InstanceData]

class mmdet.models.dense_heads.SSDHead(num_classes: int = 80, in_channels: Sequence[int] = (512, 1024, 512, 256, 256, 256), stacked_convs: int = 0, feat_channels: int = 256, use_depthwise: bool = False, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, act_cfg: Optional[Union[ConfigDict, dict]] = None, anchor_generator: Union[ConfigDict, dict] = {'basesize_ratio_range': (0.1, 0.9), 'input_size': 300, 'ratios': ([2], [2, 3], [2, 3], [2, 3], [2], [2]), 'scale_major': False, 'strides': [8, 16, 32, 64, 100, 300], 'type': 'SSDAnchorGenerator'}, bbox_coder: Union[ConfigDict, dict] = {'clip_border': True, 'target_means': [0.0, 0.0, 0.0, 0.0], 'target_stds': [1.0, 1.0, 1.0, 1.0], 'type': 'DeltaXYWHBBoxCoder'}, reg_decoded_bbox: bool = False, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'bias': 0, 'distribution': 'uniform', 'layer': 'Conv2d', 'type': 'Xavier'})

Implementation of SSD head

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (Sequence[int]) – Number of channels in the input feature map.

• stacked_convs (int) – Number of conv layers in cls and reg tower. Defaults to 0.

• feat_channels (int) – Number of hidden channels when stacked_convs > 0. Defaults to 256.

• use_depthwise (bool) – Whether to use DepthwiseSeparableConv. Defaults to False.

• conv_cfg (ConfigDict or dict, Optional) – Dictionary to construct and config conv layer. Defaults to None.

• norm_cfg (ConfigDict or dict, Optional) – Dictionary to construct and config norm layer. Defaults to None.

• act_cfg (ConfigDict or dict, Optional) – Dictionary to construct and config activation layer. Defaults to None.

• anchor_generator (ConfigDict or dict) – Config dict for anchor generator.

• bbox_coder (ConfigDict or dict) – Config of bounding box coder.

• reg_decoded_bbox (bool) – If true, the regression loss would be applied directly on decoded bounding boxes, converting both the predicted boxes and regression targets to absolute coordinates format. Defaults to False. It should be True when using IoULoss, GIoULoss, or DIoULoss in the bbox head.

• train_cfg (ConfigDict or dict, Optional) – Training config of anchor head.

• test_cfg (ConfigDict or dict, Optional) – Testing config of anchor head.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], Optional) – Initialization config dict.

forward(x: Tuple[Tensor]) → Tuple[List[Tensor], List[Tensor]]

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of cls_scores list and bbox_preds list.

• cls_scores (list[Tensor]): Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_anchors * num_classes.

• bbox_preds (list[Tensor]): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_anchors * 4.

Return type

tuple[list[Tensor], list[Tensor]]

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, List[Tensor]]

Compute losses of the head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components. the dict has components below:

• loss_cls (list[Tensor]): A list containing each feature map classification loss.

• loss_bbox (list[Tensor]): A list containing each feature map regression loss.

Return type

dict[str, list[Tensor]]

loss_by_feat_single(cls_score: Tensor, bbox_pred: Tensor, anchor: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, bbox_weights: Tensor, avg_factor: int) → Tuple[Tensor, Tensor]

Compute loss of a single image.

Parameters

• cls_score (Tensor) – Box scores for eachimage Has shape (num_total_anchors, num_classes).

• bbox_pred (Tensor) – Box energies / deltas for each image level with shape (num_total_anchors, 4).

• anchors (Tensor) – Box reference for each scale level with shape (num_total_anchors, 4).

• labels (Tensor) – Labels of each anchors with shape (num_total_anchors,).

• label_weights (Tensor) – Label weights of each anchor with shape (num_total_anchors,)

• bbox_targets (Tensor) – BBox regression targets of each anchor with shape (num_total_anchors, 4).

• bbox_weights (Tensor) – BBox regression loss weights of each anchor with shape (num_total_anchors, 4).

• avg_factor (int) – Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Returns

A tuple of cls loss and bbox loss of one feature map.

Return type

Tuple[Tensor, Tensor]

class mmdet.models.dense_heads.StageCascadeRPNHead(in_channels: int, anchor_generator: Union[ConfigDict, dict] = {'ratios': [1.0], 'scales': [8], 'strides': [4, 8, 16, 32, 64], 'type': 'AnchorGenerator'}, adapt_cfg: Union[ConfigDict, dict] = {'dilation': 3, 'type': 'dilation'}, bridged_feature: bool = False, with_cls: bool = True, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, **kwargs)

Stage of CascadeRPNHead.

Parameters

• in_channels (int) – Number of channels in the input feature map.

• anchor_generator (ConfigDict or dict) – anchor generator config.

• adapt_cfg (ConfigDict or dict) – adaptation config.

• bridged_feature (bool) – whether update rpn feature. Defaults to False.

• with_cls (bool) – whether use classification branch. Defaults to True.

:param init_cfg ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

anchor_offset(anchor_list: List[List[Tensor]], anchor_strides: List[int], featmap_sizes: List[Tuple[int, int]]) → List[Tensor]

Get offset for deformable conv based on anchor shape NOTE: currently support deformable kernel_size=3 and dilation=1

Parameters

• anchor_list (list[list[tensor])) – [NI, NLVL, NA, 4] list of multi-level anchors

• anchor_strides (list[int]) – anchor stride of each level

Returns

offset of DeformConv kernel with shapes of [NLVL, NA, 2, 18].

Return type

list[tensor]

forward(feats: List[Tensor], offset_list: Optional[List[Tensor]] = None) → Tuple[List[Tensor]]

Forward function.

forward_single(x: Tensor, offset: Tensor) → Tuple[Tensor]

Forward function of single scale.

get_targets(anchor_list: List[List[Tensor]], valid_flag_list: List[List[Tensor]], featmap_sizes: List[Tuple[int, int]], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, return_sampling_results: bool = False) → tuple

Compute regression and classification targets for anchors.

Parameters

• anchor_list (list[list[Tensor]]) – Multi level anchors of each image.

• valid_flag_list (list[list[Tensor]]) – Multi level valid flags of each image.

• featmap_sizes (list[Tuple[int, int]]) – Feature map size each level.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• return_sampling_results (bool) – Whether to return the sampling results. Defaults to False.

Returns

• labels_list (list[Tensor]): Labels of each level.

• label_weights_list (list[Tensor]): Label weights of each level.

• bbox_targets_list (list[Tensor]): BBox targets of each level.

• bbox_weights_list (list[Tensor]): BBox weights of each level.

• avg_factor (int): Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Return type

tuple

loss(x: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

loss_and_predict(x: Tuple[Tensor], batch_data_samples: List[DetDataSample], proposal_cfg: Optional[ConfigDict] = None) → Tuple[dict, List[InstanceData]]

Perform forward propagation of the head, then calculate loss and predictions from the features and data samples.

Parameters

• x (tuple[Tensor]) – Features from FPN.

• batch_data_samples (list[DetDataSample]) – Each item contains the meta information of each image and corresponding annotations.

• ( (proposal_cfg) – obj`ConfigDict`, optional): Test / postprocessing configuration, if None, test_cfg would be used. Defaults to None.

Returns

the return value is a tuple contains:

• losses: (dict[str, Tensor]): A dictionary of loss components.

• predictions (list[InstanceData]): Detection results of each image after the post process.

Return type

tuple

loss_by_feat(anchor_list: List[List[Tensor]], valid_flag_list: List[List[Tensor]], cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → Dict[str, Tensor]

Compute losses of the head.

Parameters

• anchor_list (list[list[Tensor]]) – Multi level anchors of each image.

• valid_flag_list (list[list[Tensor]]) – Multi level valid flags of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_anchors, )

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_by_feat_single(cls_score: Tensor, bbox_pred: Tensor, anchors: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, bbox_weights: Tensor, avg_factor: int) → tuple

Loss function on single scale.

predict(x: Tuple[Tensor], batch_data_samples: List[DetDataSample], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the detection head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Multi-level features from the upstream network, each is a 4D-tensor.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool, optional) – Whether to rescale the results. Defaults to False.

Returns

InstanceData]: Detection results of each image after the post process.

Return type

list[obj

predict_by_feat(anchor_list: List[List[Tensor]], cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_img_metas: List[dict], cfg: Optional[ConfigDict] = None, rescale: bool = False) → List[InstanceData]

Get proposal predict. Overriding to enable input anchor_list from outside.

Parameters

• anchor_list (list[list[Tensor]]) – Multi level anchors of each image.

• cls_scores (list[Tensor]) – Classification scores for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * 4, H, W).

• batch_img_metas (list[dict], Optional) – Image meta info.

• cfg (ConfigDict, optional) – Test / postprocessing configuration, if None, test_cfg would be used.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

refine_bboxes(anchor_list: List[List[Tensor]], bbox_preds: List[Tensor], img_metas: List[dict]) → List[List[Tensor]]

Refine bboxes through stages.

region_targets(anchor_list: List[List[Tensor]], valid_flag_list: List[List[Tensor]], featmap_sizes: List[Tuple[int, int]], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, return_sampling_results: bool = False) → tuple

Compute regression and classification targets for anchors when using RegionAssigner.

Parameters

• anchor_list (list[list[Tensor]]) – Multi level anchors of each image.

• valid_flag_list (list[list[Tensor]]) – Multi level valid flags of each image.

• featmap_sizes (list[Tuple[int, int]]) – Feature map size each level.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

• labels_list (list[Tensor]): Labels of each level.

• label_weights_list (list[Tensor]): Label weights of each level.

• bbox_targets_list (list[Tensor]): BBox targets of each level.

• bbox_weights_list (list[Tensor]): BBox weights of each level.

• avg_factor (int): Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Return type

tuple

class mmdet.models.dense_heads.TOODHead(num_classes: int, in_channels: int, num_dcn: int = 0, anchor_type: str = 'anchor_free', initial_loss_cls: Union[ConfigDict, dict] = {'activated': True, 'alpha': 0.25, 'gamma': 2.0, 'loss_weight': 1.0, 'type': 'FocalLoss', 'use_sigmoid': True}, **kwargs)

TOODHead used in `TOOD: Task-aligned One-stage Object Detection.

<https://arxiv.org/abs/2108.07755>`_.

TOOD uses Task-aligned head (T-head) and is optimized by Task Alignment Learning (TAL).

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• num_dcn (int) – Number of deformable convolution in the head. Defaults to 0.

• anchor_type (str) – If set to anchor_free, the head will use centers to regress bboxes. If set to anchor_based, the head will regress bboxes based on anchors. Defaults to anchor_free.

• initial_loss_cls (ConfigDict or dict) – Config of initial loss.

Example

>>> self = TOODHead(11, 7)
>>> feats = [torch.rand(1, 7, s, s) for s in [4, 8, 16, 32, 64]]
>>> cls_score, bbox_pred = self.forward(feats)
>>> assert len(cls_score) == len(self.scales)

anchor_center(anchors: Tensor) → Tensor

Get anchor centers from anchors.

Parameters

anchors (Tensor) – Anchor list with shape (N, 4), “xyxy” format.

Returns

Anchor centers with shape (N, 2), “xy” format.

Return type

Tensor

deform_sampling(feat: Tensor, offset: Tensor) → Tensor

Sampling the feature x according to offset.

Parameters

• feat (Tensor) – Feature

• offset (Tensor) – Spatial offset for feature sampling

forward(feats: Tuple[Tensor]) → Tuple[List[Tensor]]

Forward features from the upstream network.

Parameters

feats (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Usually a tuple of classification scores and bbox prediction

cls_scores (list[Tensor]): Classification scores for all scale

levels, each is a 4D-tensor, the channels number is num_anchors * num_classes.

bbox_preds (list[Tensor]): Decoded box for all scale levels,

each is a 4D-tensor, the channels number is num_anchors * 4. In [tl_x, tl_y, br_x, br_y] format.

Return type

tuple

get_targets(cls_scores: List[List[Tensor]], bbox_preds: List[List[Tensor]], anchor_list: List[List[Tensor]], valid_flag_list: List[List[Tensor]], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs: bool = True) → tuple

Compute regression and classification targets for anchors in multiple images.

Parameters

• cls_scores (list[list[Tensor]]) – Classification predictions of images, a 3D-Tensor with shape [num_imgs, num_priors, num_classes].

• bbox_preds (list[list[Tensor]]) – Decoded bboxes predictions of one image, a 3D-Tensor with shape [num_imgs, num_priors, 4] in [tl_x, tl_y, br_x, br_y] format.

• anchor_list (list[list[Tensor]]) – Multi level anchors of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_anchors, 4).

• valid_flag_list (list[list[Tensor]]) – Multi level valid flags of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_anchors, )

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• unmap_outputs (bool) – Whether to map outputs back to the original set of anchors.

Returns

a tuple containing learning targets.

• anchors_list (list[list[Tensor]]): Anchors of each level.

• labels_list (list[Tensor]): Labels of each level.

• label_weights_list (list[Tensor]): Label weights of each level.

• bbox_targets_list (list[Tensor]): BBox targets of each level.

• norm_alignment_metrics_list (list[Tensor]): Normalized alignment metrics of each level.

Return type

tuple

init_weights() → None

Initialize weights of the head.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Decoded box for each scale level with shape (N, num_anchors * 4, H, W) in [tl_x, tl_y, br_x, br_y] format.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

loss_by_feat_single(anchors: Tensor, cls_score: Tensor, bbox_pred: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, alignment_metrics: Tensor, stride: Tuple[int, int]) → dict

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• anchors (Tensor) – Box reference for each scale level with shape (N, num_total_anchors, 4).

• cls_score (Tensor) – Box scores for each scale level Has shape (N, num_anchors * num_classes, H, W).

• bbox_pred (Tensor) – Decoded bboxes for each scale level with shape (N, num_anchors * 4, H, W).

• labels (Tensor) – Labels of each anchors with shape (N, num_total_anchors).

• label_weights (Tensor) – Label weights of each anchor with shape (N, num_total_anchors).

• bbox_targets (Tensor) – BBox regression targets of each anchor with shape (N, num_total_anchors, 4).

• alignment_metrics (Tensor) – Alignment metrics with shape (N, num_total_anchors).

• stride (Tuple[int, int]) – Downsample stride of the feature map.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.dense_heads.VFNetHead(num_classes: int, in_channels: int, regress_ranges: Sequence[Tuple[int, int]] = ((-1, 64), (64, 128), (128, 256), (256, 512), (512, 100000000.0)), center_sampling: bool = False, center_sample_radius: float = 1.5, sync_num_pos: bool = True, gradient_mul: float = 0.1, bbox_norm_type: str = 'reg_denom', loss_cls_fl: Union[ConfigDict, dict] = {'alpha': 0.25, 'gamma': 2.0, 'loss_weight': 1.0, 'type': 'FocalLoss', 'use_sigmoid': True}, use_vfl: bool = True, loss_cls: Union[ConfigDict, dict] = {'alpha': 0.75, 'gamma': 2.0, 'iou_weighted': True, 'loss_weight': 1.0, 'type': 'VarifocalLoss', 'use_sigmoid': True}, loss_bbox: Union[ConfigDict, dict] = {'loss_weight': 1.5, 'type': 'GIoULoss'}, loss_bbox_refine: Union[ConfigDict, dict] = {'loss_weight': 2.0, 'type': 'GIoULoss'}, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 32, 'requires_grad': True, 'type': 'GN'}, use_atss: bool = True, reg_decoded_bbox: bool = True, anchor_generator: Union[ConfigDict, dict] = {'center_offset': 0.0, 'octave_base_scale': 8, 'ratios': [1.0], 'scales_per_octave': 1, 'strides': [8, 16, 32, 64, 128], 'type': 'AnchorGenerator'}, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'override': {'bias_prob': 0.01, 'name': 'vfnet_cls', 'std': 0.01, 'type': 'Normal'}, 'std': 0.01, 'type': 'Normal'}, **kwargs)

Head of `VarifocalNet (VFNet): An IoU-aware Dense Object Detector.<https://arxiv.org/abs/2008.13367>`_.

The VFNet predicts IoU-aware classification scores which mix the object presence confidence and object localization accuracy as the detection score. It is built on the FCOS architecture and uses ATSS for defining positive/negative training examples. The VFNet is trained with Varifocal Loss and empolys star-shaped deformable convolution to extract features for a bbox.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• regress_ranges (Sequence[Tuple[int, int]]) – Regress range of multiple level points.

• center_sampling (bool) – If true, use center sampling. Defaults to False.

• center_sample_radius (float) – Radius of center sampling. Defaults to 1.5.

• sync_num_pos (bool) – If true, synchronize the number of positive examples across GPUs. Defaults to True

• gradient_mul (float) – The multiplier to gradients from bbox refinement and recognition. Defaults to 0.1.

• bbox_norm_type (str) – The bbox normalization type, ‘reg_denom’ or ‘stride’. Defaults to reg_denom

• loss_cls_fl (ConfigDict or dict) – Config of focal loss.

• use_vfl (bool) – If true, use varifocal loss for training. Defaults to True.

• loss_cls (ConfigDict or dict) – Config of varifocal loss.

• loss_bbox (ConfigDict or dict) – Config of localization loss, GIoU Loss.

• loss_bbox – Config of localization refinement loss, GIoU Loss.

• norm_cfg (ConfigDict or dict) – dictionary to construct and config norm layer. Defaults to norm_cfg=dict(type=’GN’, num_groups=32, requires_grad=True).

• use_atss (bool) – If true, use ATSS to define positive/negative examples. Defaults to True.

• anchor_generator (ConfigDict or dict) – Config of anchor generator for ATSS.

:param init_cfg (ConfigDict or dict or list[dict] or: list[ConfigDict]): Initialization config dict.

Example

>>> self = VFNetHead(11, 7)
>>> feats = [torch.rand(1, 7, s, s) for s in [4, 8, 16, 32, 64]]
>>> cls_score, bbox_pred, bbox_pred_refine= self.forward(feats)
>>> assert len(cls_score) == len(self.scales)

forward(x: Tuple[Tensor]) → Tuple[List[Tensor]]

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

• cls_scores (list[Tensor]): Box iou-aware scores for each scale level, each is a 4D-tensor, the channel number is num_points * num_classes.

• bbox_preds (list[Tensor]): Box offsets for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

• bbox_preds_refine (list[Tensor]): Refined Box offsets for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

Return type

tuple

forward_single(x: Tensor, scale: Scale, scale_refine: Scale, stride: int, reg_denom: int) → tuple

Forward features of a single scale level.

Parameters

• x (Tensor) – FPN feature maps of the specified stride.

• ( (scale_refine) – obj: mmcv.cnn.Scale): Learnable scale module to resize the bbox prediction.

• ( – obj: mmcv.cnn.Scale): Learnable scale module to resize the refined bbox prediction.

• stride (int) – The corresponding stride for feature maps, used to normalize the bbox prediction when bbox_norm_type = ‘stride’.

• reg_denom (int) – The corresponding regression range for feature maps, only used to normalize the bbox prediction when bbox_norm_type = ‘reg_denom’.

Returns

iou-aware cls scores for each box, bbox predictions and refined bbox predictions of input feature maps.

Return type

tuple

get_anchors(featmap_sizes: List[Tuple], batch_img_metas: List[dict], device: str = 'cuda') → tuple

Get anchors according to feature map sizes.

Parameters

• featmap_sizes (list[tuple]) – Multi-level feature map sizes.

• batch_img_metas (list[dict]) – Image meta info.

• device (str) – Device for returned tensors

Returns

• anchor_list (list[Tensor]): Anchors of each image.

• valid_flag_list (list[Tensor]): Valid flags of each image.

Return type

tuple

get_atss_targets(cls_scores: List[Tensor], mlvl_points: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → tuple

A wrapper for computing ATSS targets for points in multiple images.

Parameters

• cls_scores (list[Tensor]) – Box iou-aware scores for each scale level with shape (N, num_points * num_classes, H, W).

• mlvl_points (list[Tensor]) – Points of each fpn level, each has shape (num_points, 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

• labels_list (list[Tensor]): Labels of each level.

• label_weights (Tensor): Label weights of all levels.

• bbox_targets_list (list[Tensor]): Regression targets of each level, (l, t, r, b).

• bbox_weights (Tensor): Bbox weights of all levels.

Return type

tuple

get_fcos_targets(points: List[Tensor], batch_gt_instances: List[InstanceData]) → tuple

Compute FCOS regression and classification targets for points in multiple images.

Parameters

• points (list[Tensor]) – Points of each fpn level, each has shape (num_points, 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

Returns

• labels (list[Tensor]): Labels of each level.

• label_weights: None, to be compatible with ATSS targets.

• bbox_targets (list[Tensor]): BBox targets of each level.

• bbox_weights: None, to be compatible with ATSS targets.

Return type

tuple

get_targets(cls_scores: List[Tensor], mlvl_points: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → tuple

A wrapper for computing ATSS and FCOS targets for points in multiple images.

Parameters

• cls_scores (list[Tensor]) – Box iou-aware scores for each scale level with shape (N, num_points * num_classes, H, W).

• mlvl_points (list[Tensor]) – Points of each fpn level, each has shape (num_points, 2).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

• labels_list (list[Tensor]): Labels of each level.

• label_weights (Tensor/None): Label weights of all levels.

• bbox_targets_list (list[Tensor]): Regression targets of each level, (l, t, r, b).

• bbox_weights (Tensor/None): Bbox weights of all levels.

Return type

tuple

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], bbox_preds_refine: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Compute loss of the head.

Parameters

• cls_scores (list[Tensor]) – Box iou-aware scores for each scale level, each is a 4D-tensor, the channel number is num_points * num_classes.

• bbox_preds (list[Tensor]) – Box offsets for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

• bbox_preds_refine (list[Tensor]) – Refined Box offsets for each scale level, each is a 4D-tensor, the channel number is num_points * 4.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], Optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

star_dcn_offset(bbox_pred: Tensor, gradient_mul: float, stride: int) → Tensor

Compute the star deformable conv offsets.

Parameters

• bbox_pred (Tensor) – Predicted bbox distance offsets (l, r, t, b).

• gradient_mul (float) – Gradient multiplier.

• stride (int) – The corresponding stride for feature maps, used to project the bbox onto the feature map.

Returns

The offsets for deformable convolution.

Return type

Tensor

transform_bbox_targets(decoded_bboxes: List[Tensor], mlvl_points: List[Tensor], num_imgs: int) → List[Tensor]

Transform bbox_targets (x1, y1, x2, y2) into (l, t, r, b) format.

Parameters

• decoded_bboxes (list[Tensor]) – Regression targets of each level, in the form of (x1, y1, x2, y2).

• mlvl_points (list[Tensor]) – Points of each fpn level, each has shape (num_points, 2).

• num_imgs (int) – the number of images in a batch.

Returns

Regression targets of each level in

the form of (l, t, r, b).

Return type

bbox_targets (list[Tensor])

class mmdet.models.dense_heads.YOLACTHead(num_classes: int, in_channels: int, anchor_generator: Union[ConfigDict, dict] = {'octave_base_scale': 3, 'ratios': [0.5, 1.0, 2.0], 'scales_per_octave': 1, 'strides': [8, 16, 32, 64, 128], 'type': 'AnchorGenerator'}, loss_cls: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'reduction': 'none', 'type': 'CrossEntropyLoss', 'use_sigmoid': False}, loss_bbox: Union[ConfigDict, dict] = {'beta': 1.0, 'loss_weight': 1.5, 'type': 'SmoothL1Loss'}, num_head_convs: int = 1, num_protos: int = 32, use_ohem: bool = True, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = {'bias': 0, 'distribution': 'uniform', 'layer': 'Conv2d', 'type': 'Xavier'}, **kwargs)

YOLACT box head used in https://arxiv.org/abs/1904.02689.

Note that YOLACT head is a light version of RetinaNet head. Four differences are described as follows:

1. YOLACT box head has three-times fewer anchors.

2. YOLACT box head shares the convs for box and cls branches.

3. YOLACT box head uses OHEM instead of Focal loss.

4. YOLACT box head predicts a set of mask coefficients for each box.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• anchor_generator (ConfigDict or dict) – Config dict for anchor generator

• loss_cls (ConfigDict or dict) – Config of classification loss.

• loss_bbox (ConfigDict or dict) – Config of localization loss.

• num_head_convs (int) – Number of the conv layers shared by box and cls branches.

• num_protos (int) – Number of the mask coefficients.

• use_ohem (bool) – If true, loss_single_OHEM will be used for cls loss calculation. If false, loss_single will be used.

• conv_cfg (ConfigDict or dict, optional) – Dictionary to construct and config conv layer.

• norm_cfg (ConfigDict or dict, optional) – Dictionary to construct and config norm layer.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

OHEMloss_by_feat_single(cls_score: Tensor, bbox_pred: Tensor, anchors: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, bbox_weights: Tensor, avg_factor: int) → tuple

Compute loss of a single image. Similar to func:SSDHead.loss_by_feat_single

Parameters

• cls_score (Tensor) – Box scores for eachimage Has shape (num_total_anchors, num_classes).

• bbox_pred (Tensor) – Box energies / deltas for each image level with shape (num_total_anchors, 4).

• anchors (Tensor) – Box reference for each scale level with shape (num_total_anchors, 4).

• labels (Tensor) – Labels of each anchors with shape (num_total_anchors,).

• label_weights (Tensor) – Label weights of each anchor with shape (num_total_anchors,)

• bbox_targets (Tensor) – BBox regression targets of each anchor with shape (num_total_anchors, 4).

• bbox_weights (Tensor) – BBox regression loss weights of each anchor with shape (num_total_anchors, 4).

• avg_factor (int) – Average factor that is used to average the loss. When using sampling method, avg_factor is usually the sum of positive and negative priors. When using PseudoSampler, avg_factor is usually equal to the number of positive priors.

Returns

A tuple of cls loss and bbox loss of one feature map.

Return type

Tuple[Tensor, Tensor]

forward_single(x: Tensor) → tuple

Forward feature of a single scale level.

Parameters

x (Tensor) – Features of a single scale level.

Returns

• cls_score (Tensor): Cls scores for a single scale level the channels number is num_anchors * num_classes.

• bbox_pred (Tensor): Box energies / deltas for a single scale level, the channels number is num_anchors * 4.

• coeff_pred (Tensor): Mask coefficients for a single scale level, the channels number is num_anchors * num_protos.

Return type

tuple

get_positive_infos() → List[InstanceData]

Get positive information from sampling results.

Returns

Positive Information of each image, usually including positive bboxes, positive labels, positive priors, positive coeffs, etc.

Return type

list[InstanceData]

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], coeff_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the bbox head.

When self.use_ohem == True, it functions like SSDHead.loss, otherwise, it follows AnchorHead.loss.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level has shape (N, num_anchors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• coeff_preds (list[Tensor]) – Mask coefficients for each scale level with shape (N, num_anchors * num_protos, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict

predict_by_feat(cls_scores, bbox_preds, coeff_preds, batch_img_metas, cfg=None, rescale=True, **kwargs)

Similar to func:AnchorHead.get_bboxes, but additionally processes coeff_preds.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level with shape (N, num_anchors * num_classes, H, W)

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W)

• coeff_preds (list[Tensor]) – Mask coefficients for each scale level with shape (N, num_anchors * num_protos, H, W)

• batch_img_metas (list[dict]) – Batch image meta info.

• cfg (Config | None) – Test / postprocessing configuration, if None, test_cfg would be used

• rescale (bool) – If True, return boxes in original image space. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• coeffs (Tensor): the predicted mask coefficients of instance inside the corresponding box has a shape (n, num_protos).

Return type

list[InstanceData]

class mmdet.models.dense_heads.YOLACTProtonet(num_classes: int, in_channels: int = 256, proto_channels: tuple = (256, 256, 256, None, 256, 32), proto_kernel_sizes: tuple = (3, 3, 3, -2, 3, 1), include_last_relu: bool = True, num_protos: int = 32, loss_mask_weight: float = 1.0, max_masks_to_train: int = 100, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, with_seg_branch: bool = True, loss_segm: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, init_cfg={'distribution': 'uniform', 'override': {'name': 'protonet'}, 'type': 'Xavier'})

YOLACT mask head used in https://arxiv.org/abs/1904.02689.

This head outputs the mask prototypes for YOLACT.

Parameters

• in_channels (int) – Number of channels in the input feature map.

• proto_channels (tuple[int]) – Output channels of protonet convs.

• proto_kernel_sizes (tuple[int]) – Kernel sizes of protonet convs.

• include_last_relu (bool) – If keep the last relu of protonet.

• num_protos (int) – Number of prototypes.

• num_classes (int) – Number of categories excluding the background category.

• loss_mask_weight (float) – Reweight the mask loss by this factor.

• max_masks_to_train (int) – Maximum number of masks to train for each image.

• with_seg_branch (bool) – Whether to apply a semantic segmentation branch and calculate loss during training to increase performance with no speed penalty. Defaults to True.

• loss_segm (ConfigDict or dict, optional) – Config of semantic segmentation loss.

• train_cfg (ConfigDict or dict, optional) – Training config of head.

• test_cfg (ConfigDict or dict, optional) – Testing config of head.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

crop_mask_preds(mask_preds: List[Tensor], batch_img_metas: List[dict], positive_infos: List[InstanceData]) → list

Crop predicted masks by zeroing out everything not in the predicted bbox.

Parameters

• mask_preds (list[Tensor]) – Predicted prototypes with shape (num_classes, H, W).

• batch_img_metas (list[dict]) – Meta information of multiple images.

• positive_infos (List[:obj:InstanceData]) – Positive information that calculate from detect head.

Returns

The cropped masks.

Return type

list

crop_single(masks: Tensor, boxes: Tensor, padding: int = 1) → Tensor

Crop single predicted masks by zeroing out everything not in the predicted bbox.

Parameters

• masks (Tensor) – Predicted prototypes, has shape [H, W, N].

• boxes (Tensor) – Bbox coords in relative point form with shape [N, 4].

• padding (int) – Image padding size.

Returns

The cropped masks.

Return type

Tensor

forward(x: tuple, positive_infos: List[InstanceData]) → tuple

Forward feature from the upstream network to get prototypes and linearly combine the prototypes, using masks coefficients, into instance masks. Finally, crop the instance masks with given bboxes.

Parameters

• x (Tuple[Tensor]) – Feature from the upstream network, which is a 4D-tensor.

• positive_infos (List[:obj:InstanceData]) – Positive information that calculate from detect head.

Returns

Predicted instance segmentation masks and semantic segmentation map.

Return type

tuple

loss_by_feat(mask_preds: List[Tensor], segm_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], positive_infos: List[InstanceData], **kwargs) → dict

Calculate the loss based on the features extracted by the mask head.

Parameters

• mask_preds (list[Tensor]) – List of predicted prototypes, each has shape (num_classes, H, W).

• segm_preds (Tensor) – Predicted semantic segmentation map with shape (N, num_classes, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, masks, and labels attributes.

• batch_img_metas (list[dict]) – Meta information of multiple images.

• positive_infos (List[:obj:InstanceData]) – Information of positive samples of each image that are assigned in detection head.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

predict_by_feat(mask_preds: List[Tensor], segm_preds: Tensor, results_list: List[InstanceData], batch_img_metas: List[dict], rescale: bool = True, **kwargs) → List[InstanceData]

Transform a batch of output features extracted from the head into mask results.

Parameters

• mask_preds (list[Tensor]) – Predicted prototypes with shape (num_classes, H, W).

• results_list (List[:obj:InstanceData]) – BBoxHead results.

• batch_img_metas (list[dict]) – Meta information of all images.

• rescale (bool, optional) – Whether to rescale the results. Defaults to False.

Returns

Processed results of multiple images.Each InstanceData usually contains following keys.

• scores (Tensor): Classification scores, has shape (num_instance,).

• labels (Tensor): Has shape (num_instances,).

• masks (Tensor): Processed mask results, has shape (num_instances, h, w).

Return type

list[InstanceData]

sanitize_coordinates(x1: Tensor, x2: Tensor, img_size: int, padding: int = 0, cast: bool = True) → tuple

Sanitizes the input coordinates so that x1 < x2, x1 != x2, x1 >= 0, and x2 <= image_size. Also converts from relative to absolute coordinates and casts the results to long tensors.

Warning: this does things in-place behind the scenes so copy if necessary.

Parameters

• x1 (Tensor) – shape (N, ).

• x2 (Tensor) – shape (N, ).

• img_size (int) – Size of the input image.

• padding (int) – x1 >= padding, x2 <= image_size-padding.

• cast (bool) – If cast is false, the result won’t be cast to longs.

Returns

• x1 (Tensor): Sanitized _x1.

• x2 (Tensor): Sanitized _x2.

Return type

tuple

class mmdet.models.dense_heads.YOLOFHead(num_classes: int, in_channels: List[int], num_cls_convs: int = 2, num_reg_convs: int = 4, norm_cfg: Union[ConfigDict, dict] = {'requires_grad': True, 'type': 'BN'}, **kwargs)

Detection Head of YOLOF

Parameters

• num_classes (int) – The number of object classes (w/o background)

• in_channels (list[int]) – The number of input channels per scale.

• cls_num_convs (int) – The number of convolutions of cls branch. Defaults to 2.

• reg_num_convs (int) – The number of convolutions of reg branch. Defaults to 4.

• norm_cfg (ConfigDict or dict) – Config dict for normalization layer. Defaults to dict(type='BN', requires_grad=True).

forward_single(x: Tensor) → Tuple[Tensor, Tensor]

Forward feature of a single scale level.

Parameters

x (Tensor) – Features of a single scale level.

Returns

normalized_cls_score (Tensor): Normalized Cls scores for a single scale level, the channels number is num_base_priors * num_classes. bbox_reg (Tensor): Box energies / deltas for a single scale level, the channels number is num_base_priors * 4.

Return type

tuple

get_targets(cls_scores_list: List[Tensor], bbox_preds_list: List[Tensor], anchor_list: List[Tensor], valid_flag_list: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None, unmap_outputs: bool = True)

Compute regression and classification targets for anchors in multiple images.

Parameters

• cls_scores_list (list[Tensor]) – Classification scores of each image. each is a 4D-tensor, the shape is (h * w, num_anchors * num_classes).

• bbox_preds_list (list[Tensor]) – Bbox preds of each image. each is a 4D-tensor, the shape is (h * w, num_anchors * 4).

• anchor_list (list[Tensor]) – Anchors of each image. Each element of is a tensor of shape (h * w * num_anchors, 4).

• valid_flag_list (list[Tensor]) – Valid flags of each image. Each element of is a tensor of shape (h * w * num_anchors, )

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• unmap_outputs (bool) – Whether to map outputs back to the original set of anchors.

Returns

Usually returns a tuple containing learning targets.

• batch_labels (Tensor): Label of all images. Each element of is a tensor of shape (batch, h * w * num_anchors)

• batch_label_weights (Tensor): Label weights of all images of is a tensor of shape (batch, h * w * num_anchors)

• num_total_pos (int): Number of positive samples in all images.

• num_total_neg (int): Number of negative samples in all images.

additional_returns: This function enables user-defined returns from

self._get_targets_single. These returns are currently refined to properties at each feature map (i.e. having HxW dimension). The results will be concatenated after the end

Return type

tuple

init_weights() → None

Initialize the weights.

loss_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (list[Tensor]) – Box scores for each scale level has shape (N, num_anchors * num_classes, H, W).

• bbox_preds (list[Tensor]) – Box energies / deltas for each scale level with shape (N, num_anchors * 4, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict

class mmdet.models.dense_heads.YOLOV3Head(num_classes: int, in_channels: Sequence[int], out_channels: Sequence[int] = (1024, 512, 256), anchor_generator: Union[ConfigDict, dict] = {'base_sizes': [[(116, 90), (156, 198), (373, 326)], [(30, 61), (62, 45), (59, 119)], [(10, 13), (16, 30), (33, 23)]], 'strides': [32, 16, 8], 'type': 'YOLOAnchorGenerator'}, bbox_coder: Union[ConfigDict, dict] = {'type': 'YOLOBBoxCoder'}, featmap_strides: Sequence[int] = (32, 16, 8), one_hot_smoother: float = 0.0, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'requires_grad': True, 'type': 'BN'}, act_cfg: Union[ConfigDict, dict] = {'negative_slope': 0.1, 'type': 'LeakyReLU'}, loss_cls: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_conf: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_xy: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_wh: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'MSELoss'}, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None)

YOLOV3Head Paper link: https://arxiv.org/abs/1804.02767.

Parameters

• num_classes (int) – The number of object classes (w/o background)

• in_channels (Sequence[int]) – Number of input channels per scale.

• out_channels (Sequence[int]) – The number of output channels per scale before the final 1x1 layer. Default: (1024, 512, 256).

• anchor_generator (ConfigDict or dict) – Config dict for anchor generator.

• bbox_coder (ConfigDict or dict) – Config of bounding box coder.

• featmap_strides (Sequence[int]) – The stride of each scale. Should be in descending order. Defaults to (32, 16, 8).

• one_hot_smoother (float) – Set a non-zero value to enable label-smooth Defaults to 0.

• conv_cfg (ConfigDict or dict, optional) – Config dict for convolution layer. Defaults to None.

• norm_cfg (ConfigDict or dict) – Dictionary to construct and config norm layer. Defaults to dict(type=’BN’, requires_grad=True).

• act_cfg (ConfigDict or dict) – Config dict for activation layer. Defaults to dict(type=’LeakyReLU’, negative_slope=0.1).

• loss_cls (ConfigDict or dict) – Config of classification loss.

• loss_conf (ConfigDict or dict) – Config of confidence loss.

• loss_xy (ConfigDict or dict) – Config of xy coordinate loss.

• loss_wh (ConfigDict or dict) – Config of wh coordinate loss.

• train_cfg (ConfigDict or dict, optional) – Training config of YOLOV3 head. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – Testing config of YOLOV3 head. Defaults to None.

forward(x: Tuple[Tensor, ...]) → tuple

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of multi-level predication map, each is a

4D-tensor of shape (batch_size, 5+num_classes, height, width).

Return type

tuple[Tensor]

get_targets(anchor_list: List[List[Tensor]], responsible_flag_list: List[List[Tensor]], batch_gt_instances: List[InstanceData]) → tuple

Compute target maps for anchors in multiple images.

Parameters

• anchor_list (list[list[Tensor]]) – Multi level anchors of each image. The outer list indicates images, and the inner list corresponds to feature levels of the image. Each element of the inner list is a tensor of shape (num_total_anchors, 4).

• responsible_flag_list (list[list[Tensor]]) – Multi level responsible flags of each image. Each element is a tensor of shape (num_total_anchors, )

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

Returns

Usually returns a tuple containing learning targets.

• target_map_list (list[Tensor]): Target map of each level.

• neg_map_list (list[Tensor]): Negative map of each level.

Return type

tuple

init_weights() → None

Initialize weights.

loss_by_feat(pred_maps: Sequence[Tensor], batch_gt_instances: List[InstanceData], batch_img_metas: List[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• pred_maps (list[Tensor]) – Prediction map for each scale level, shape (N, num_anchors * num_attrib, H, W)

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of loss components.

Return type

dict

loss_by_feat_single(pred_map: Tensor, target_map: Tensor, neg_map: Tensor) → tuple

Calculate the loss of a single scale level based on the features extracted by the detection head.

Parameters

• pred_map (Tensor) – Raw predictions for a single level.

• target_map (Tensor) – The Ground-Truth target for a single level.

• neg_map (Tensor) – The negative masks for a single level.

Returns

loss_cls (Tensor): Classification loss. loss_conf (Tensor): Confidence loss. loss_xy (Tensor): Regression loss of x, y coordinate. loss_wh (Tensor): Regression loss of w, h coordinate.

Return type

tuple

property num_attrib: int

number of attributes in pred_map, bboxes (4) + objectness (1) + num_classes

Type

int

property num_levels: int

number of feature map levels

Type

int

predict_by_feat(pred_maps: Sequence[Tensor], batch_img_metas: Optional[List[dict]], cfg: Optional[Union[ConfigDict, dict]] = None, rescale: bool = False, with_nms: bool = True) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results. It has been accelerated since PR #5991.

Parameters

• pred_maps (Sequence[Tensor]) – Raw predictions for a batch of images.

• batch_img_metas (list[dict], Optional) – Batch image meta info. Defaults to None.

• cfg (ConfigDict or dict, optional) – Test / postprocessing configuration, if None, test_cfg would be used. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

• with_nms (bool) – If True, do nms before return boxes. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

responsible_flags(featmap_sizes: List[tuple], gt_bboxes: Tensor, device: str) → List[Tensor]

Generate responsible anchor flags of grid cells in multiple scales.

Parameters

• featmap_sizes (List[tuple]) – List of feature map sizes in multiple feature levels.

• gt_bboxes (Tensor) – Ground truth boxes, shape (n, 4).

• device (str) – Device where the anchors will be put on.

Returns

responsible flags of anchors in multiple level

Return type

List[Tensor]

class mmdet.models.dense_heads.YOLOXHead(num_classes: int, in_channels: int, feat_channels: int = 256, stacked_convs: int = 2, strides: Sequence[int] = (8, 16, 32), use_depthwise: bool = False, dcn_on_last_conv: bool = False, conv_bias: Union[bool, str] = 'auto', conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'eps': 0.001, 'momentum': 0.03, 'type': 'BN'}, act_cfg: Union[ConfigDict, dict] = {'type': 'Swish'}, loss_cls: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'reduction': 'sum', 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_bbox: Union[ConfigDict, dict] = {'eps': 1e-16, 'loss_weight': 5.0, 'mode': 'square', 'reduction': 'sum', 'type': 'IoULoss'}, loss_obj: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'reduction': 'sum', 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_l1: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'reduction': 'sum', 'type': 'L1Loss'}, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = {'a': 2.23606797749979, 'distribution': 'uniform', 'layer': 'Conv2d', 'mode': 'fan_in', 'nonlinearity': 'leaky_relu', 'type': 'Kaiming'})

YOLOXHead head used in YOLOX.

Parameters

• num_classes (int) – Number of categories excluding the background category.

• in_channels (int) – Number of channels in the input feature map.

• feat_channels (int) – Number of hidden channels in stacking convs. Defaults to 256

• stacked_convs (int) – Number of stacking convs of the head. Defaults to (8, 16, 32).

• strides (Sequence[int]) – Downsample factor of each feature map. Defaults to None.

• use_depthwise (bool) – Whether to depthwise separable convolution in blocks. Defaults to False.

• dcn_on_last_conv (bool) – If true, use dcn in the last layer of towers. Defaults to False.

• conv_bias (bool or str) – If specified as auto, it will be decided by the norm_cfg. Bias of conv will be set as True if norm_cfg is None, otherwise False. Defaults to “auto”.

• conv_cfg (ConfigDict or dict, optional) – Config dict for convolution layer. Defaults to None.

• norm_cfg (ConfigDict or dict) – Config dict for normalization layer. Defaults to dict(type=’BN’, momentum=0.03, eps=0.001).

• act_cfg (ConfigDict or dict) – Config dict for activation layer. Defaults to None.

• loss_cls (ConfigDict or dict) – Config of classification loss.

• loss_bbox (ConfigDict or dict) – Config of localization loss.

• loss_obj (ConfigDict or dict) – Config of objectness loss.

• loss_l1 (ConfigDict or dict) – Config of L1 loss.

• train_cfg (ConfigDict or dict, optional) – Training config of anchor head. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – Testing config of anchor head. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

forward(x: Tuple[Tensor]) → Tuple[List]

Forward features from the upstream network.

Parameters

x (Tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of multi-level classification scores, bbox predictions, and objectnesses.

Return type

Tuple[List]

forward_single(x: Tensor, cls_convs: Module, reg_convs: Module, conv_cls: Module, conv_reg: Module, conv_obj: Module) → Tuple[Tensor, Tensor, Tensor]

Forward feature of a single scale level.

init_weights() → None

Initialize weights of the head.

loss_by_feat(cls_scores: Sequence[Tensor], bbox_preds: Sequence[Tensor], objectnesses: Sequence[Tensor], batch_gt_instances: Sequence[InstanceData], batch_img_metas: Sequence[dict], batch_gt_instances_ignore: Optional[List[InstanceData]] = None) → dict

Calculate the loss based on the features extracted by the detection head.

Parameters

• cls_scores (Sequence[Tensor]) – Box scores for each scale level, each is a 4D-tensor, the channel number is num_priors * num_classes.

• bbox_preds (Sequence[Tensor]) – Box energies / deltas for each scale level, each is a 4D-tensor, the channel number is num_priors * 4.

• objectnesses (Sequence[Tensor]) – Score factor for all scale level, each is a 4D-tensor, has shape (batch_size, 1, H, W).

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes and labels attributes.

• batch_img_metas (list[dict]) – Meta information of each image, e.g., image size, scaling factor, etc.

• batch_gt_instances_ignore (list[InstanceData], optional) – Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

Returns

A dictionary of losses.

Return type

dict[str, Tensor]

predict_by_feat(cls_scores: List[Tensor], bbox_preds: List[Tensor], objectnesses: Optional[List[Tensor]], batch_img_metas: Optional[List[dict]] = None, cfg: Optional[ConfigDict] = None, rescale: bool = False, with_nms: bool = True) → List[InstanceData]

Transform a batch of output features extracted by the head into bbox results. :param cls_scores: Classification scores for all

scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * num_classes, H, W).

Parameters

• bbox_preds (list[Tensor]) – Box energies / deltas for all scale levels, each is a 4D-tensor, has shape (batch_size, num_priors * 4, H, W).

• objectnesses (list[Tensor], Optional) – Score factor for all scale level, each is a 4D-tensor, has shape (batch_size, 1, H, W).

• batch_img_metas (list[dict], Optional) – Batch image meta info. Defaults to None.

• cfg (ConfigDict, optional) – Test / postprocessing configuration, if None, test_cfg would be used. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

• with_nms (bool) – If True, do nms before return boxes. Defaults to True.

Returns

Object detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

detectors

class mmdet.models.detectors.ATSS(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of ATSS

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• train_cfg (ConfigDict or dict, optional) – The training config of ATSS. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of ATSS. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

class mmdet.models.detectors.AutoAssign(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of AutoAssign: Differentiable Label Assignment for Dense Object Detection

Parameters

• backbone (ConfigDict or dict) – The backbone config.

• neck (ConfigDict or dict) – The neck config.

• bbox_head (ConfigDict or dict) – The bbox head config.

• train_cfg (ConfigDict or dict, optional) – The training config of AutoAssign. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of AutoAssign. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

class mmdet.models.detectors.BaseDetector(data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Base class for detectors.

Parameters

• data_preprocessor (dict or ConfigDict, optional) –

  The pre-process config of BaseDataPreprocessor. it usually includes,

  pad_size_divisor, pad_value, mean and std.

• init_cfg (dict or ConfigDict, optional) – the config to control the initialization. Defaults to None.

add_pred_to_datasample(data_samples: List[DetDataSample], results_list: List[InstanceData]) → List[DetDataSample]

Add predictions to DetDataSample.

Parameters

• data_samples (list[DetDataSample], optional) – A batch of data samples that contain annotations and predictions.

• results_list (list[InstanceData]) – Detection results of each image.

Returns

Detection results of the input images. Each DetDataSample usually contain ‘pred_instances’. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape

  (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape

  (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4),

  the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[DetDataSample]

abstract extract_feat(batch_inputs: Tensor)

Extract features from images.

forward(inputs: Tensor, data_samples: Optional[List[DetDataSample]] = None, mode: str = 'tensor') → Union[Dict[str, Tensor], List[DetDataSample], Tuple[Tensor], Tensor]

The unified entry for a forward process in both training and test.

The method should accept three modes: “tensor”, “predict” and “loss”:

• “tensor”: Forward the whole network and return tensor or tuple of

tensor without any post-processing, same as a common nn.Module. - “predict”: Forward and return the predictions, which are fully processed to a list of DetDataSample. - “loss”: Forward and return a dict of losses according to the given inputs and data samples.

Note that this method doesn’t handle either back propagation or parameter update, which are supposed to be done in train_step().

Parameters

• inputs (torch.Tensor) – The input tensor with shape (N, C, …) in general.

• data_samples (list[DetDataSample], optional) – A batch of data samples that contain annotations and predictions. Defaults to None.

• mode (str) – Return what kind of value. Defaults to ‘tensor’.

Returns

The return type depends on mode.

• If mode="tensor", return a tensor or a tuple of tensor.

• If mode="predict", return a list of DetDataSample.

• If mode="loss", return a dict of tensor.

abstract loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → Union[dict, tuple]

Calculate losses from a batch of inputs and data samples.

abstract predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → List[DetDataSample]

Predict results from a batch of inputs and data samples with post- processing.

property with_bbox: bool

whether the detector has a bbox head

Type

bool

property with_mask: bool

whether the detector has a mask head

Type

bool

property with_neck: bool

whether the detector has a neck

Type

bool

property with_shared_head: bool

whether the detector has a shared head in the RoI Head

Type

bool

class mmdet.models.detectors.BoxInst(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], mask_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of BoxInst

class mmdet.models.detectors.CLRNet(backbone, neck=None, head=None, data_preprocessor=None)

class mmdet.models.detectors.CascadeRCNN(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, rpn_head: Optional[Union[ConfigDict, dict]] = None, roi_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Cascade R-CNN: Delving into High Quality Object Detection

class mmdet.models.detectors.CenterNet(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of CenterNet(Objects as Points)

<https://arxiv.org/abs/1904.07850>.

class mmdet.models.detectors.CondInst(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], mask_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of CondInst

class mmdet.models.detectors.ConditionalDETR(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, encoder: Optional[Union[ConfigDict, dict]] = None, decoder: Optional[Union[ConfigDict, dict]] = None, bbox_head: Optional[Union[ConfigDict, dict]] = None, positional_encoding: Optional[Union[ConfigDict, dict]] = None, num_queries: int = 100, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of `Conditional DETR for Fast Training Convergence.

<https://arxiv.org/abs/2108.06152>`_.

Code is modified from the official github repo.

forward_decoder(query: Tensor, query_pos: Tensor, memory: Tensor, memory_mask: Tensor, memory_pos: Tensor) → Dict

Forward with Transformer decoder.

Parameters

• query (Tensor) – The queries of decoder inputs, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional queries of decoder inputs, has shape (bs, num_queries, dim).

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points).

• memory_pos (Tensor) – The positional embeddings of memory, has shape (bs, num_feat_points, dim).

Returns

The dictionary of decoder outputs, which includes the hidden_states and references of the decoder output.

• hidden_states (Tensor): Has shape

  (num_decoder_layers, bs, num_queries, dim)

• references (Tensor): Has shape

  (bs, num_queries, 2)

Return type

dict

class mmdet.models.detectors.CornerNet(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

CornerNet.

This detector is the implementation of the paper CornerNet: Detecting Objects as Paired Keypoints .

class mmdet.models.detectors.CrowdDet(backbone: Union[ConfigDict, dict], rpn_head: Union[ConfigDict, dict], roi_head: Union[ConfigDict, dict], train_cfg: Union[ConfigDict, dict], test_cfg: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of CrowdDet

Parameters

• backbone (ConfigDict or dict) – The backbone config.

• rpn_head (ConfigDict or dict) – The rpn config.

• roi_head (ConfigDict or dict) – The roi config.

• train_cfg (ConfigDict or dict, optional) – The training config of FCOS. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of FCOS. Defaults to None.

• neck (ConfigDict or dict) – The neck config.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

class mmdet.models.detectors.DABDETR(*args, with_random_refpoints: bool = False, num_patterns: int = 0, **kwargs)

Implementation of `DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR.

<https://arxiv.org/abs/2201.12329>`_.

Code is modified from the official github repo.

Parameters

• with_random_refpoints (bool) – Whether to randomly initialize query embeddings and not update them during training. Defaults to False.

• num_patterns (int) – Inspired by Anchor-DETR. Defaults to 0.

forward_decoder(query: Tensor, query_pos: Tensor, memory: Tensor, memory_mask: Tensor, memory_pos: Tensor) → Dict

Forward with Transformer decoder.

Parameters

• query (Tensor) – The queries of decoder inputs, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional queries of decoder inputs, has shape (bs, num_queries, dim).

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points).

• memory_pos (Tensor) – The positional embeddings of memory, has shape (bs, num_feat_points, dim).

Returns

The dictionary of decoder outputs, which includes the hidden_states and references of the decoder output.

Return type

dict

init_weights() → None

Initialize weights for Transformer and other components.

pre_decoder(memory: Tensor) → Tuple[Dict, Dict]

Prepare intermediate variables before entering Transformer decoder, such as query, query_pos.

Parameters

memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

Returns

The first dict contains the inputs of decoder and the second dict contains the inputs of the bbox_head function.

• decoder_inputs_dict (dict): The keyword args dictionary of

  self.forward_decoder(), which includes ‘query’, ‘query_pos’, ‘memory’ and ‘reg_branches’.

• head_inputs_dict (dict): The keyword args dictionary of the

  bbox_head functions, which is usually empty, or includes enc_outputs_class and enc_outputs_class when the detector support ‘two stage’ or ‘query selection’ strategies.

Return type

tuple[dict, dict]

class mmdet.models.detectors.DDOD(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of DDOD.

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• train_cfg (ConfigDict or dict, optional) – The training config of ATSS. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of ATSS. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

class mmdet.models.detectors.DDQDETR(*args, dense_topk_ratio: float = 1.5, dqs_cfg: Optional[Union[ConfigDict, dict]] = {'iou_threshold': 0.8, 'type': 'nms'}, **kwargs)

Implementation of Dense Distinct Query for End-to-End Object Detection

Code is modified from the official github repo.

Parameters

• dense_topk_ratio (float) – Ratio of num_dense queries to num_queries. Defaults to 1.5.

• dqs_cfg (ConfigDict or dict, optional) – Config of Distinct Queries Selection. Defaults to nms with iou_threshold = 0.8.

init_weights() → None

Initialize weights for Transformer and other components.

pre_decoder(memory: Tensor, memory_mask: Tensor, spatial_shapes: Tensor, batch_data_samples: Optional[List[DetDataSample]] = None) → Tuple[Dict]

Prepare intermediate variables before entering Transformer decoder, such as query, memory, and reference_points.

Parameters

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points). Will only be used when as_two_stage is True.

• spatial_shapes (Tensor) – Spatial shapes of features in all levels. With shape (num_levels, 2), last dimension represents (h, w). Will only be used when as_two_stage is True.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg. Defaults to None.

Returns

The decoder_inputs_dict and head_inputs_dict.

• decoder_inputs_dict (dict): The keyword dictionary args of self.forward_decoder(), which includes ‘query’, ‘memory’, reference_points, and dn_mask. The reference points of decoder input here are 4D boxes, although it has points in its name.

• head_inputs_dict (dict): The keyword dictionary args of the bbox_head functions, which includes topk_score, topk_coords, dense_topk_score, dense_topk_coords, and dn_meta, when self.training is True, else is empty.

Return type

tuple[dict]

class mmdet.models.detectors.DETR(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, encoder: Optional[Union[ConfigDict, dict]] = None, decoder: Optional[Union[ConfigDict, dict]] = None, bbox_head: Optional[Union[ConfigDict, dict]] = None, positional_encoding: Optional[Union[ConfigDict, dict]] = None, num_queries: int = 100, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of `DETR: End-to-End Object Detection with Transformers.

<https://arxiv.org/pdf/2005.12872>`_.

Code is modified from the official github repo.

forward_decoder(query: Tensor, query_pos: Tensor, memory: Tensor, memory_mask: Tensor, memory_pos: Tensor) → Dict

Forward with Transformer decoder.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

• query (Tensor) – The queries of decoder inputs, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional queries of decoder inputs, has shape (bs, num_queries, dim).

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points).

• memory_pos (Tensor) – The positional embeddings of memory, has shape (bs, num_feat_points, dim).

Returns

The dictionary of decoder outputs, which includes the hidden_states of the decoder output.

• hidden_states (Tensor): Has shape (num_decoder_layers, bs, num_queries, dim)

Return type

dict

forward_encoder(feat: Tensor, feat_mask: Tensor, feat_pos: Tensor) → Dict

Forward with Transformer encoder.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

• feat (Tensor) – Sequential features, has shape (bs, num_feat_points, dim).

• feat_mask (Tensor) – ByteTensor, the padding mask of the features, has shape (bs, num_feat_points).

• feat_pos (Tensor) – The positional embeddings of the features, has shape (bs, num_feat_points, dim).

Returns

The dictionary of encoder outputs, which includes the memory of the encoder output.

Return type

dict

init_weights() → None

Initialize weights for Transformer and other components.

pre_decoder(memory: Tensor) → Tuple[Dict, Dict]

Prepare intermediate variables before entering Transformer decoder, such as query, query_pos.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

Returns

The first dict contains the inputs of decoder and the second dict contains the inputs of the bbox_head function.

• decoder_inputs_dict (dict): The keyword args dictionary of self.forward_decoder(), which includes ‘query’, ‘query_pos’, ‘memory’.

• head_inputs_dict (dict): The keyword args dictionary of the bbox_head functions, which is usually empty, or includes enc_outputs_class and enc_outputs_class when the detector support ‘two stage’ or ‘query selection’ strategies.

Return type

tuple[dict, dict]

pre_transformer(img_feats: Tuple[Tensor], batch_data_samples: Optional[List[DetDataSample]] = None) → Tuple[Dict, Dict]

Prepare the inputs of the Transformer.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

• img_feats (Tuple[Tensor]) – Tuple of features output from the neck, has shape (bs, c, h, w).

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg. Defaults to None.

Returns

The first dict contains the inputs of encoder and the second dict contains the inputs of decoder.

• encoder_inputs_dict (dict): The keyword args dictionary of self.forward_encoder(), which includes ‘feat’, ‘feat_mask’, and ‘feat_pos’.

• decoder_inputs_dict (dict): The keyword args dictionary of self.forward_decoder(), which includes ‘memory_mask’, and ‘memory_pos’.

Return type

tuple[dict, dict]

class mmdet.models.detectors.DINO(*args, dn_cfg: Optional[Union[ConfigDict, dict]] = None, **kwargs)

Implementation of DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

Code is modified from the official github repo.

Parameters

dn_cfg (ConfigDict or dict, optional) – Config of denoising query generator. Defaults to None.

forward_decoder(query: Tensor, memory: Tensor, memory_mask: Tensor, reference_points: Tensor, spatial_shapes: Tensor, level_start_index: Tensor, valid_ratios: Tensor, dn_mask: Optional[Tensor] = None, **kwargs) → Dict

Forward with Transformer decoder.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

• query (Tensor) – The queries of decoder inputs, has shape (bs, num_queries_total, dim), where num_queries_total is the sum of num_denoising_queries and num_matching_queries when self.training is True, else num_matching_queries.

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points).

• reference_points (Tensor) – The initial reference, has shape (bs, num_queries_total, 4) with the last dimension arranged as (cx, cy, w, h).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• level_start_index (Tensor) – The start index of each level. A tensor has shape (num_levels, ) and can be represented as [0, h_0*w_0, h_0*w_0+h_1*w_1, …].

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

• dn_mask (Tensor, optional) – The attention mask to prevent information leakage from different denoising groups and matching parts, will be used as self_attn_mask of the self.decoder, has shape (num_queries_total, num_queries_total). It is None when self.training is False.

Returns

The dictionary of decoder outputs, which includes the hidden_states of the decoder output and references including the initial and intermediate reference_points.

Return type

dict

forward_transformer(img_feats: Tuple[Tensor], batch_data_samples: Optional[List[DetDataSample]] = None) → Dict

Forward process of Transformer.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py. The difference is that the ground truth in batch_data_samples is required for the pre_decoder to prepare the query of DINO. Additionally, DINO inherits the pre_transformer method and the forward_encoder method of DeformableDETR. More details about the two methods can be found in mmdet/detector/deformable_detr.py.

Parameters

• img_feats (tuple[Tensor]) – Tuple of feature maps from neck. Each feature map has shape (bs, dim, H, W).

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg. Defaults to None.

Returns

The dictionary of bbox_head function inputs, which always includes the hidden_states of the decoder output and may contain references including the initial and intermediate references.

Return type

dict

init_weights() → None

Initialize weights for Transformer and other components.

pre_decoder(memory: Tensor, memory_mask: Tensor, spatial_shapes: Tensor, batch_data_samples: Optional[List[DetDataSample]] = None) → Tuple[Dict]

Prepare intermediate variables before entering Transformer decoder, such as query, query_pos, and reference_points.

Parameters

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points). Will only be used when as_two_stage is True.

• spatial_shapes (Tensor) – Spatial shapes of features in all levels. With shape (num_levels, 2), last dimension represents (h, w). Will only be used when as_two_stage is True.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg. Defaults to None.

Returns

The decoder_inputs_dict and head_inputs_dict.

• decoder_inputs_dict (dict): The keyword dictionary args of self.forward_decoder(), which includes ‘query’, ‘memory’, reference_points, and dn_mask. The reference points of decoder input here are 4D boxes, although it has points in its name.

• head_inputs_dict (dict): The keyword dictionary args of the bbox_head functions, which includes topk_score, topk_coords, and dn_meta when self.training is True, else is empty.

Return type

tuple[dict]

class mmdet.models.detectors.DeformableDETR(*args, decoder: Optional[Union[ConfigDict, dict]] = None, bbox_head: Optional[Union[ConfigDict, dict]] = None, with_box_refine: bool = False, as_two_stage: bool = False, num_feature_levels: int = 4, **kwargs)

Implementation of Deformable DETR: Deformable Transformers for End-to-End Object Detection

Code is modified from the official github repo.

Parameters

• decoder (ConfigDict or dict, optional) – Config of the Transformer decoder. Defaults to None.

• bbox_head (ConfigDict or dict, optional) – Config for the bounding box head module. Defaults to None.

• with_box_refine (bool, optional) – Whether to refine the references in the decoder. Defaults to False.

• as_two_stage (bool, optional) – Whether to generate the proposal from the outputs of encoder. Defaults to False.

• num_feature_levels (int, optional) – Number of feature levels. Defaults to 4.

forward_decoder(query: Tensor, query_pos: Tensor, memory: Tensor, memory_mask: Tensor, reference_points: Tensor, spatial_shapes: Tensor, level_start_index: Tensor, valid_ratios: Tensor) → Dict

Forward with Transformer decoder.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

• query (Tensor) – The queries of decoder inputs, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional queries of decoder inputs, has shape (bs, num_queries, dim).

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points).

• reference_points (Tensor) – The initial reference, has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h) when as_two_stage is True, otherwise has shape (bs, num_queries, 2) with the last dimension arranged as (cx, cy).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• level_start_index (Tensor) – The start index of each level. A tensor has shape (num_levels, ) and can be represented as [0, h_0*w_0, h_0*w_0+h_1*w_1, …].

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

Returns

The dictionary of decoder outputs, which includes the hidden_states of the decoder output and references including the initial and intermediate reference_points.

Return type

dict

forward_encoder(feat: Tensor, feat_mask: Tensor, feat_pos: Tensor, spatial_shapes: Tensor, level_start_index: Tensor, valid_ratios: Tensor) → Dict

Forward with Transformer encoder.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

• feat (Tensor) – Sequential features, has shape (bs, num_feat_points, dim).

• feat_mask (Tensor) – ByteTensor, the padding mask of the features, has shape (bs, num_feat_points).

• feat_pos (Tensor) – The positional embeddings of the features, has shape (bs, num_feat_points, dim).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• level_start_index (Tensor) – The start index of each level. A tensor has shape (num_levels, ) and can be represented as [0, h_0*w_0, h_0*w_0+h_1*w_1, …].

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

Returns

The dictionary of encoder outputs, which includes the memory of the encoder output.

Return type

dict

gen_encoder_output_proposals(memory: Tensor, memory_mask: Tensor, spatial_shapes: Tensor) → Tuple[Tensor, Tensor]

Generate proposals from encoded memory. The function will only be used when as_two_stage is True.

Parameters

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

Returns

A tuple of transformed memory and proposals.

• output_memory (Tensor): The transformed memory for obtaining top-k proposals, has shape (bs, num_feat_points, dim).

• output_proposals (Tensor): The inverse-normalized proposal, has shape (batch_size, num_keys, 4) with the last dimension arranged as (cx, cy, w, h).

Return type

tuple

static get_proposal_pos_embed(proposals: Tensor, num_pos_feats: int = 128, temperature: int = 10000) → Tensor

Get the position embedding of the proposal.

Parameters

• proposals (Tensor) – Not normalized proposals, has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• num_pos_feats (int, optional) – The feature dimension for each position along x, y, w, and h-axis. Note the final returned dimension for each position is 4 times of num_pos_feats. Default to 128.

• temperature (int, optional) – The temperature used for scaling the position embedding. Defaults to 10000.

Returns

The position embedding of proposal, has shape (bs, num_queries, num_pos_feats * 4), with the last dimension arranged as (cx, cy, w, h)

Return type

Tensor

static get_valid_ratio(mask: Tensor) → Tensor

Get the valid radios of feature map in a level.

          |---> valid_W <---|
       ---+-----------------+-----+---
        A |                 |     | A
        | |                 |     | |
        | |                 |     | |
  valid_H |                 |     | |
        | |                 |     | H
        | |                 |     | |
        V |                 |     | |
       ---+-----------------+     | |
          |                       | V
          +-----------------------+---
          |---------> W <---------|

The valid_ratios are defined as:
      r_h = valid_H / H,  r_w = valid_W / W
They are the factors to re-normalize the relative coordinates of the
image to the relative coordinates of the current level feature map.

Parameters

mask (Tensor) – Binary mask of a feature map, has shape (bs, H, W).

Returns

valid ratios [r_w, r_h] of a feature map, has shape (1, 2).

Return type

Tensor

init_weights() → None

Initialize weights for Transformer and other components.

pre_decoder(memory: Tensor, memory_mask: Tensor, spatial_shapes: Tensor) → Tuple[Dict, Dict]

Prepare intermediate variables before entering Transformer decoder, such as query, query_pos, and reference_points.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points). It will only be used when as_two_stage is True.

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w). It will only be used when as_two_stage is True.

Returns

The decoder_inputs_dict and head_inputs_dict.

• decoder_inputs_dict (dict): The keyword dictionary args of self.forward_decoder(), which includes ‘query’, ‘query_pos’, ‘memory’, and reference_points. The reference_points of decoder input here are 4D boxes when as_two_stage is True, otherwise 2D points, although it has points in its name. The reference_points in encoder is always 2D points.

• head_inputs_dict (dict): The keyword dictionary args of the bbox_head functions, which includes enc_outputs_class and enc_outputs_coord. They are both None when ‘as_two_stage’ is False. The dict is empty when self.training is False.

Return type

tuple[dict, dict]

pre_transformer(mlvl_feats: Tuple[Tensor], batch_data_samples: Optional[List[DetDataSample]] = None) → Tuple[Dict]

Process image features before feeding them to the transformer.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

• mlvl_feats (tuple[Tensor]) – Multi-level features that may have different resolutions, output from neck. Each feature has shape (bs, dim, h_lvl, w_lvl), where ‘lvl’ means ‘layer’.

• batch_data_samples (list[DetDataSample], optional) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg. Defaults to None.

Returns

The first dict contains the inputs of encoder and the second dict contains the inputs of decoder.

• encoder_inputs_dict (dict): The keyword args dictionary of self.forward_encoder(), which includes ‘feat’, ‘feat_mask’, and ‘feat_pos’.

• decoder_inputs_dict (dict): The keyword args dictionary of self.forward_decoder(), which includes ‘memory_mask’.

Return type

tuple[dict]

class mmdet.models.detectors.DetectionTransformer(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, encoder: Optional[Union[ConfigDict, dict]] = None, decoder: Optional[Union[ConfigDict, dict]] = None, bbox_head: Optional[Union[ConfigDict, dict]] = None, positional_encoding: Optional[Union[ConfigDict, dict]] = None, num_queries: int = 100, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Base class for Detection Transformer.

In Detection Transformer, an encoder is used to process output features of neck, then several queries interact with the encoder features using a decoder and do the regression and classification with the bounding box head.

Parameters

• backbone (ConfigDict or dict) – Config of the backbone.

• neck (ConfigDict or dict, optional) – Config of the neck. Defaults to None.

• encoder (ConfigDict or dict, optional) – Config of the Transformer encoder. Defaults to None.

• decoder (ConfigDict or dict, optional) – Config of the Transformer decoder. Defaults to None.

• bbox_head (ConfigDict or dict, optional) – Config for the bounding box head module. Defaults to None.

• positional_encoding (ConfigDict or dict, optional) – Config of the positional encoding module. Defaults to None.

• num_queries (int, optional) – Number of decoder query in Transformer. Defaults to 100.

• train_cfg (ConfigDict or dict, optional) – Training config of the bounding box head module. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – Testing config of the bounding box head module. Defaults to None.

• data_preprocessor (dict or ConfigDict, optional) – The pre-process config of BaseDataPreprocessor. it usually includes, pad_size_divisor, pad_value, mean and std. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

extract_feat(batch_inputs: Tensor) → Tuple[Tensor]

Extract features.

Parameters

batch_inputs (Tensor) – Image tensor, has shape (bs, dim, H, W).

Returns

Tuple of feature maps from neck. Each feature map has shape (bs, dim, H, W).

Return type

tuple[Tensor]

abstract forward_decoder(query: Tensor, query_pos: Tensor, memory: Tensor, **kwargs) → Dict

Forward with Transformer decoder.

Parameters

• query (Tensor) – The queries of decoder inputs, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional queries of decoder inputs, has shape (bs, num_queries, dim).

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

Returns

The dictionary of decoder outputs, which includes the hidden_states of the decoder output, references including the initial and intermediate reference_points, and other algorithm-specific arguments.

Return type

dict

abstract forward_encoder(feat: Tensor, feat_mask: Tensor, feat_pos: Tensor, **kwargs) → Dict

Forward with Transformer encoder.

Parameters

• feat (Tensor) – Sequential features, has shape (bs, num_feat_points, dim).

• feat_mask (Tensor) – ByteTensor, the padding mask of the features, has shape (bs, num_feat_points).

• feat_pos (Tensor) – The positional embeddings of the features, has shape (bs, num_feat_points, dim).

Returns

The dictionary of encoder outputs, which includes the memory of the encoder output and other algorithm-specific arguments.

Return type

dict

forward_transformer(img_feats: Tuple[Tensor], batch_data_samples: Optional[List[DetDataSample]] = None) → Dict

Forward process of Transformer, which includes four steps: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’. We summarized the parameters flow of the existing DETR-like detector, which can be illustrated as follow:

img_feats & batch_data_samples
              |
              V
     +-----------------+
     | pre_transformer |
     +-----------------+
         |          |
         |          V
         |    +-----------------+
         |    | forward_encoder |
         |    +-----------------+
         |             |
         |             V
         |     +---------------+
         |     |  pre_decoder  |
         |     +---------------+
         |         |       |
         V         V       |
     +-----------------+   |
     | forward_decoder |   |
     +-----------------+   |
               |           |
               V           V
              head_inputs_dict

Parameters

• img_feats (tuple[Tensor]) – Tuple of feature maps from neck. Each feature map has shape (bs, dim, H, W).

• batch_data_samples (list[DetDataSample], optional) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg. Defaults to None.

Returns

The dictionary of bbox_head function inputs, which always includes the hidden_states of the decoder output and may contain references including the initial and intermediate references.

Return type

dict

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → Union[dict, list]

Calculate losses from a batch of inputs and data samples.

Parameters

• batch_inputs (Tensor) – Input images of shape (bs, dim, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict

abstract pre_decoder(memory: Tensor, **kwargs) → Tuple[Dict, Dict]

Prepare intermediate variables before entering Transformer decoder, such as query, query_pos, and reference_points.

Parameters

memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

Returns

The first dict contains the inputs of decoder and the second dict contains the inputs of the bbox_head function.

• decoder_inputs_dict (dict): The keyword dictionary args of self.forward_decoder(), which includes ‘query’, ‘query_pos’, ‘memory’, and other algorithm-specific arguments.

• head_inputs_dict (dict): The keyword dictionary args of the bbox_head functions, which is usually empty, or includes enc_outputs_class and enc_outputs_class when the detector support ‘two stage’ or ‘query selection’ strategies.

Return type

tuple[dict, dict]

abstract pre_transformer(img_feats: Tuple[Tensor], batch_data_samples: Optional[List[DetDataSample]] = None) → Tuple[Dict, Dict]

Process image features before feeding them to the transformer.

Parameters

• img_feats (tuple[Tensor]) – Tuple of feature maps from neck. Each feature map has shape (bs, dim, H, W).

• batch_data_samples (list[DetDataSample], optional) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg. Defaults to None.

Returns

The first dict contains the inputs of encoder and the second dict contains the inputs of decoder.

• encoder_inputs_dict (dict): The keyword args dictionary of self.forward_encoder(), which includes ‘feat’, ‘feat_mask’, ‘feat_pos’, and other algorithm-specific arguments.

• decoder_inputs_dict (dict): The keyword args dictionary of self.forward_decoder(), which includes ‘memory_mask’, and other algorithm-specific arguments.

Return type

tuple[dict, dict]

predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[DetDataSample]

Predict results from a batch of inputs and data samples with post- processing.

Parameters

• batch_inputs (Tensor) – Inputs, has shape (bs, dim, H, W).

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

• rescale (bool) – Whether to rescale the results. Defaults to True.

Returns

Detection results of the input images. Each DetDataSample usually contain ‘pred_instances’. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[DetDataSample]

class mmdet.models.detectors.Detectron2Wrapper(detector: Union[ConfigDict, dict], bgr_to_rgb: bool = False, rgb_to_bgr: bool = False)

Wrapper of a Detectron2 model. Input/output formats of this class follow MMDetection’s convention, so a Detectron2 model can be trained and evaluated in MMDetection.

Parameters

• detector (ConfigDict or dict) – The module config of Detectron2.

• bgr_to_rgb (bool) – whether to convert image from BGR to RGB. Defaults to False.

• rgb_to_bgr (bool) – whether to convert image from RGB to BGR. Defaults to False.

extract_feat(*args, **kwargs)

Extract features from images.

extract_feat will not be used in obj:Detectron2Wrapper.

init_weights() → None

Initialization Backbone.

NOTE: The initialization of other layers are in Detectron2, if users want to change the initialization way, please change the code in Detectron2.

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → Union[dict, tuple]

Calculate losses from a batch of inputs and data samples.

The inputs will first convert to the Detectron2 type and feed into D2 models.

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → List[DetDataSample]

Predict results from a batch of inputs and data samples with post- processing.

The inputs will first convert to the Detectron2 type and feed into D2 models. And the results will convert back to the MMDet type.

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

Detection results of the input images. Each DetDataSample usually contain ‘pred_instances’. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[DetDataSample]

class mmdet.models.detectors.FCOS(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of FCOS

Parameters

• backbone (ConfigDict or dict) – The backbone config.

• neck (ConfigDict or dict) – The neck config.

• bbox_head (ConfigDict or dict) – The bbox head config.

• train_cfg (ConfigDict or dict, optional) – The training config of FCOS. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of FCOS. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

class mmdet.models.detectors.FOVEA(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of FoveaBox :param backbone: The backbone config. :type backbone: ConfigDict or dict :param neck: The neck config. :type neck: ConfigDict or dict :param bbox_head: The bbox head config. :type bbox_head: ConfigDict or dict :param train_cfg: The training config

of FOVEA. Defaults to None.

Parameters

• test_cfg (ConfigDict or dict, optional) – The testing config of FOVEA. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

class mmdet.models.detectors.FSAF(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of FSAF

class mmdet.models.detectors.FastRCNN(backbone: Union[ConfigDict, dict], roi_head: Union[ConfigDict, dict], train_cfg: Union[ConfigDict, dict], test_cfg: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Fast R-CNN

class mmdet.models.detectors.FasterRCNN(backbone: Union[ConfigDict, dict], rpn_head: Union[ConfigDict, dict], roi_head: Union[ConfigDict, dict], train_cfg: Union[ConfigDict, dict], test_cfg: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Faster R-CNN

class mmdet.models.detectors.GFL(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of GFL

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• train_cfg (ConfigDict or dict, optional) – The training config of GFL. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of GFL. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

class mmdet.models.detectors.GLIP(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], language_model: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of GLIP :param backbone: The backbone config. :type backbone: ConfigDict or dict :param neck: The neck config. :type neck: ConfigDict or dict :param bbox_head: The bbox head config. :type bbox_head: ConfigDict or dict :param language_model: The language model config. :type language_model: ConfigDict or dict :param train_cfg: The training config

of GLIP. Defaults to None.

Parameters

• test_cfg (ConfigDict or dict, optional) – The testing config of GLIP. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

get_tokens_and_prompts(original_caption: Union[str, list, tuple], custom_entities: bool = False, enhanced_text_prompts: Optional[Union[ConfigDict, dict]] = None) → Tuple[dict, str, list, list]

Get the tokens positive and prompts for the caption.

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → Union[dict, list]

Calculate losses from a batch of inputs and data samples.

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[DetDataSample]

Predict results from a batch of inputs and data samples with post- processing.

Parameters

• batch_inputs (Tensor) – Inputs with shape (N, C, H, W).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results. Defaults to True.

Returns

Detection results of the input images. Each DetDataSample usually contain ‘pred_instances’. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape

  (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape

  (num_instances, ).

• label_names (List[str]): Label names of bboxes.

• bboxes (Tensor): Has a shape (num_instances, 4),

  the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[DetDataSample]

class mmdet.models.detectors.GridRCNN(backbone: Union[ConfigDict, dict], rpn_head: Union[ConfigDict, dict], roi_head: Union[ConfigDict, dict], train_cfg: Union[ConfigDict, dict], test_cfg: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Grid R-CNN.

This detector is the implementation of: - Grid R-CNN (https://arxiv.org/abs/1811.12030) - Grid R-CNN Plus: Faster and Better (https://arxiv.org/abs/1906.05688)

class mmdet.models.detectors.GroundingDINO(language_model, *args, use_autocast=False, **kwargs)

Implementation of `Grounding DINO: Marrying DINO with Grounded Pre- Training for Open-Set Object Detection.

<https://arxiv.org/abs/2303.05499>`_

Code is modified from the official github repo.

forward_encoder(feat: Tensor, feat_mask: Tensor, feat_pos: Tensor, spatial_shapes: Tensor, level_start_index: Tensor, valid_ratios: Tensor, text_dict: Dict) → Dict

Forward with Transformer encoder.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py.

Parameters

• feat (Tensor) – Sequential features, has shape (bs, num_feat_points, dim).

• feat_mask (Tensor) – ByteTensor, the padding mask of the features, has shape (bs, num_feat_points).

• feat_pos (Tensor) – The positional embeddings of the features, has shape (bs, num_feat_points, dim).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• level_start_index (Tensor) – The start index of each level. A tensor has shape (num_levels, ) and can be represented as [0, h_0*w_0, h_0*w_0+h_1*w_1, …].

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

Returns

The dictionary of encoder outputs, which includes the memory of the encoder output.

Return type

dict

forward_transformer(img_feats: Tuple[Tensor], text_dict: Dict, batch_data_samples: Optional[List[DetDataSample]] = None) → Dict

Forward process of Transformer.

The forward procedure of the transformer is defined as: ‘pre_transformer’ -> ‘encoder’ -> ‘pre_decoder’ -> ‘decoder’ More details can be found at TransformerDetector.forward_transformer in mmdet/detector/base_detr.py. The difference is that the ground truth in batch_data_samples is required for the pre_decoder to prepare the query of DINO. Additionally, DINO inherits the pre_transformer method and the forward_encoder method of DeformableDETR. More details about the two methods can be found in mmdet/detector/deformable_detr.py.

Parameters

• img_feats (tuple[Tensor]) – Tuple of feature maps from neck. Each feature map has shape (bs, dim, H, W).

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg. Defaults to None.

Returns

The dictionary of bbox_head function inputs, which always includes the hidden_states of the decoder output and may contain references including the initial and intermediate references.

Return type

dict

get_tokens_and_prompts(original_caption: Union[str, list, tuple], custom_entities: bool = False, enhanced_text_prompts: Optional[Union[ConfigDict, dict]] = None) → Tuple[dict, str, list]

Get the tokens positive and prompts for the caption.

get_tokens_positive_and_prompts(original_caption: Union[str, list, tuple], custom_entities: bool = False, enhanced_text_prompt: Optional[Union[ConfigDict, dict]] = None, tokens_positive: Optional[list] = None) → Tuple[dict, str, Tensor, list]

Get the tokens positive and prompts for the caption.

Parameters

• original_caption (str) – The original caption, e.g. ‘bench . car .’

• custom_entities (bool, optional) – Whether to use custom entities. If True, the original_caption should be a list of strings, each of which is a word. Defaults to False.

Returns

The dict is a mapping from each entity id, which is numbered from 1, to its positive token id. The str represents the prompts.

Return type

Tuple[dict, str, dict, str]

init_weights() → None

Initialize weights for Transformer and other components.

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → Union[dict, list]

Calculate losses from a batch of inputs and data samples.

Parameters

• batch_inputs (Tensor) – Input images of shape (bs, dim, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict

pre_decoder(memory: Tensor, memory_mask: Tensor, spatial_shapes: Tensor, memory_text: Tensor, text_token_mask: Tensor, batch_data_samples: Optional[List[DetDataSample]] = None) → Tuple[Dict]

Prepare intermediate variables before entering Transformer decoder, such as query, query_pos, and reference_points.

Parameters

• memory (Tensor) – The output embeddings of the Transformer encoder, has shape (bs, num_feat_points, dim).

• memory_mask (Tensor) – ByteTensor, the padding mask of the memory, has shape (bs, num_feat_points). Will only be used when as_two_stage is True.

• spatial_shapes (Tensor) – Spatial shapes of features in all levels. With shape (num_levels, 2), last dimension represents (h, w). Will only be used when as_two_stage is True.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg. Defaults to None.

Returns

The decoder_inputs_dict and head_inputs_dict.

• decoder_inputs_dict (dict): The keyword dictionary args of self.forward_decoder(), which includes ‘query’, ‘memory’, reference_points, and dn_mask. The reference points of decoder input here are 4D boxes, although it has points in its name.

• head_inputs_dict (dict): The keyword dictionary args of the bbox_head functions, which includes topk_score, topk_coords, and dn_meta when self.training is True, else is empty.

Return type

tuple[dict]

predict(batch_inputs, batch_data_samples, rescale: bool = True)

Predict results from a batch of inputs and data samples with post- processing.

Parameters

• batch_inputs (Tensor) – Inputs, has shape (bs, dim, H, W).

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

• rescale (bool) – Whether to rescale the results. Defaults to True.

Returns

Detection results of the input images. Each DetDataSample usually contain ‘pred_instances’. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[DetDataSample]

class mmdet.models.detectors.HybridTaskCascade(**kwargs)

Implementation of HTC

property with_semantic: bool

whether the detector has a semantic head

Type

bool

class mmdet.models.detectors.KnowledgeDistillationSingleStageDetector(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], teacher_config: Union[ConfigDict, dict, str, Path], teacher_ckpt: Optional[str] = None, eval_teacher: bool = True, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None)

Implementation of Distilling the Knowledge in a Neural Network..

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• teacher_config (ConfigDict | dict | str | Path) – Config file path or the config object of teacher model.

• teacher_ckpt (str, optional) – Checkpoint path of teacher model. If left as None, the model will not load any weights. Defaults to True.

• eval_teacher (bool) – Set the train mode for teacher. Defaults to True.

• train_cfg (ConfigDict or dict, optional) – The training config of ATSS. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of ATSS. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

cuda(device: Optional[str] = None) → Module

Since teacher_model is registered as a plain object, it is necessary to put the teacher model to cuda when calling cuda function.

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → dict

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

to(device: Optional[str] = None) → Module

Since teacher_model is registered as a plain object, it is necessary to put the teacher model to other device when calling to function.

train(mode: bool = True) → None

Set the same train mode for teacher and student model.

class mmdet.models.detectors.LAD(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], teacher_backbone: Union[ConfigDict, dict], teacher_neck: Union[ConfigDict, dict], teacher_bbox_head: Union[ConfigDict, dict], teacher_ckpt: Optional[str] = None, eval_teacher: bool = True, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None)

Implementation of LAD.

extract_teacher_feat(batch_inputs: Tensor) → Tensor

Directly extract teacher features from the backbone+neck.

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → dict

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

property with_teacher_neck: bool

whether the detector has a teacher_neck

Type

bool

class mmdet.models.detectors.Mask2Former(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, panoptic_head: Optional[Union[ConfigDict, dict]] = None, panoptic_fusion_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Masked-attention Mask Transformer for Universal Image Segmentation.

class mmdet.models.detectors.MaskFormer(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, panoptic_head: Optional[Union[ConfigDict, dict]] = None, panoptic_fusion_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Per-Pixel Classification is NOT All You Need for Semantic Segmentation.

add_pred_to_datasample(data_samples: List[DetDataSample], results_list: List[dict]) → List[DetDataSample]

Add predictions to DetDataSample.

Parameters

• data_samples (list[DetDataSample], optional) – A batch of data samples that contain annotations and predictions.

• results_list (List[dict]) – Instance segmentation, segmantic segmentation and panoptic segmentation results.

Returns

Detection results of the input images. Each DetDataSample usually contain ‘pred_instances’ and pred_panoptic_seg. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape

  (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape

  (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4),

  the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

And the pred_panoptic_seg contains the following key

• sem_seg (Tensor): panoptic segmentation mask, has a

  shape (1, h, w).

Return type

list[DetDataSample]

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → Dict[str, Tensor]

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

a dictionary of loss components

Return type

dict[str, Tensor]

predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[DetDataSample]

Predict results from a batch of inputs and data samples with post- processing.

Parameters

• batch_inputs (Tensor) – Inputs with shape (N, C, H, W).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results. Defaults to True.

Returns

Detection results of the input images. Each DetDataSample usually contain ‘pred_instances’ and pred_panoptic_seg. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape

  (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape

  (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4),

  the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

And the pred_panoptic_seg contains the following key

• sem_seg (Tensor): panoptic segmentation mask, has a

  shape (1, h, w).

Return type

list[DetDataSample]

class mmdet.models.detectors.MaskRCNN(backbone: ConfigDict, rpn_head: ConfigDict, roi_head: ConfigDict, train_cfg: ConfigDict, test_cfg: ConfigDict, neck: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Mask R-CNN

class mmdet.models.detectors.MaskScoringRCNN(backbone: Union[ConfigDict, dict], rpn_head: Union[ConfigDict, dict], roi_head: Union[ConfigDict, dict], train_cfg: Union[ConfigDict, dict], test_cfg: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Mask Scoring RCNN.

https://arxiv.org/abs/1903.00241

class mmdet.models.detectors.NASFCOS(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of NAS-FCOS: Fast Neural Architecture Search for Object Detection.

Parameters

• backbone (ConfigDict or dict) – The backbone config.

• neck (ConfigDict or dict) – The neck config.

• bbox_head (ConfigDict or dict) – The bbox head config.

• train_cfg (ConfigDict or dict, optional) – The training config of NASFCOS. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of NASFCOS. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

class mmdet.models.detectors.PAA(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of PAA

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• train_cfg (ConfigDict or dict, optional) – The training config of PAA. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of PAA. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

class mmdet.models.detectors.PanopticFPN(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, rpn_head: Optional[Union[ConfigDict, dict]] = None, roi_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, semantic_head: Optional[Union[ConfigDict, dict]] = None, panoptic_fusion_head: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Panoptic feature pyramid networks

class mmdet.models.detectors.PointRend(backbone: ConfigDict, rpn_head: ConfigDict, roi_head: ConfigDict, train_cfg: ConfigDict, test_cfg: ConfigDict, neck: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

PointRend: Image Segmentation as Rendering

This detector is the implementation of PointRend.

class mmdet.models.detectors.QueryInst(backbone: Union[ConfigDict, dict], rpn_head: Union[ConfigDict, dict], roi_head: Union[ConfigDict, dict], train_cfg: Union[ConfigDict, dict], test_cfg: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Instances as Queries

class mmdet.models.detectors.RPN(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], rpn_head: Union[ConfigDict, dict], train_cfg: Union[ConfigDict, dict], test_cfg: Union[ConfigDict, dict], data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, **kwargs)

Implementation of Region Proposal Network.

Parameters

• backbone (ConfigDict or dict) – The backbone config.

• neck (ConfigDict or dict) – The neck config.

• bbox_head (ConfigDict or dict) – The bbox head config.

• train_cfg (ConfigDict or dict, optional) – The training config.

• test_cfg (ConfigDict or dict, optional) – The testing config.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → dict

Calculate losses from a batch of inputs and data samples.

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict[str, Tensor]

class mmdet.models.detectors.RTMDet(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, use_syncbn: bool = True)

Implementation of RTMDet.

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• train_cfg (ConfigDict or dict, optional) – The training config of ATSS. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of ATSS. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

• use_syncbn (bool) – Whether to use SyncBatchNorm. Defaults to True.

class mmdet.models.detectors.RepPointsDetector(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

RepPoints: Point Set Representation for Object Detection.

This detector is the implementation of: - RepPoints detector (https://arxiv.org/pdf/1904.11490)

class mmdet.models.detectors.RetinaNet(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of RetinaNet

class mmdet.models.detectors.SCNet(**kwargs)

Implementation of SCNet

class mmdet.models.detectors.SOLO(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, bbox_head: Optional[Union[ConfigDict, dict]] = None, mask_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

SOLO: Segmenting Objects by Locations

class mmdet.models.detectors.SOLOv2(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, bbox_head: Optional[Union[ConfigDict, dict]] = None, mask_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

SOLOv2: Dynamic and Fast Instance Segmentation

class mmdet.models.detectors.SemiBaseDetector(detector: Union[ConfigDict, dict], semi_train_cfg: Optional[Union[ConfigDict, dict]] = None, semi_test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Base class for semi-supervised detectors.

Semi-supervised detectors typically consisting of a teacher model updated by exponential moving average and a student model updated by gradient descent.

Parameters

• detector (ConfigDict or dict) – The detector config.

• semi_train_cfg (ConfigDict or dict, optional) – The semi-supervised training config.

• semi_test_cfg (ConfigDict or dict, optional) – The semi-supervised testing config.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

extract_feat(batch_inputs: Tensor) → Tuple[Tensor]

Extract features.

Parameters

batch_inputs (Tensor) – Image tensor with shape (N, C, H ,W).

Returns

Multi-level features that may have different resolutions.

Return type

tuple[Tensor]

static freeze(model: Module)

Freeze the model.

get_pseudo_instances(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → Tuple[List[DetDataSample], Optional[dict]]

Get pseudo instances from teacher model.

loss(multi_batch_inputs: Dict[str, Tensor], multi_batch_data_samples: Dict[str, List[DetDataSample]]) → dict

Calculate losses from multi-branch inputs and data samples.

Parameters

• multi_batch_inputs (Dict[str, Tensor]) – The dict of multi-branch input images, each value with shape (N, C, H, W). Each value should usually be mean centered and std scaled.

• multi_batch_data_samples (Dict[str, List[DetDataSample]]) – The dict of multi-branch data samples.

Returns

A dictionary of loss components

Return type

dict

loss_by_gt_instances(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → dict

Calculate losses from a batch of inputs and ground-truth data samples.

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict

loss_by_pseudo_instances(batch_inputs: Tensor, batch_data_samples: List[DetDataSample], batch_info: Optional[dict] = None) → dict

Calculate losses from a batch of inputs and pseudo data samples.

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg, which are pseudo_instance or pseudo_panoptic_seg or pseudo_sem_seg in fact.

• batch_info (dict) – Batch information of teacher model forward propagation process. Defaults to None.

Returns

A dictionary of loss components

Return type

dict

predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → List[DetDataSample]

Predict results from a batch of inputs and data samples with post- processing.

Parameters

• batch_inputs (Tensor) – Inputs with shape (N, C, H, W).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results. Defaults to True.

Returns

Return the detection results of the input images. The returns value is DetDataSample, which usually contain ‘pred_instances’. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape

  (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape

  (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4),

  the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[DetDataSample]

project_pseudo_instances(batch_pseudo_instances: List[DetDataSample], batch_data_samples: List[DetDataSample]) → List[DetDataSample]

Project pseudo instances.

class mmdet.models.detectors.SingleStageDetector(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, bbox_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Base class for single-stage detectors.

Single-stage detectors directly and densely predict bounding boxes on the output features of the backbone+neck.

extract_feat(batch_inputs: Tensor) → Tuple[Tensor]

Extract features.

Parameters

batch_inputs (Tensor) – Image tensor with shape (N, C, H ,W).

Returns

Multi-level features that may have different resolutions.

Return type

tuple[Tensor]

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → Union[dict, list]

Calculate losses from a batch of inputs and data samples.

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[DetDataSample]

Predict results from a batch of inputs and data samples with post- processing.

Parameters

• batch_inputs (Tensor) – Inputs with shape (N, C, H, W).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results. Defaults to True.

Returns

Detection results of the input images. Each DetDataSample usually contain ‘pred_instances’. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape

  (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape

  (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4),

  the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[DetDataSample]

class mmdet.models.detectors.SoftTeacher(detector: Union[ConfigDict, dict], semi_train_cfg: Optional[Union[ConfigDict, dict]] = None, semi_test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of End-to-End Semi-Supervised Object Detection with Soft Teacher

Parameters

• detector (ConfigDict or dict) – The detector config.

• semi_train_cfg (ConfigDict or dict, optional) – The semi-supervised training config.

• semi_test_cfg (ConfigDict or dict, optional) – The semi-supervised testing config.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

static aug_box(batch_data_samples, times, frac)

Augment bboxes with jitter.

compute_uncertainty_with_aug(x: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → List[Tensor]

Compute uncertainty with augmented bboxes.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg, which are pseudo_instance or pseudo_panoptic_seg or pseudo_sem_seg in fact.

Returns

A list of uncertainty for pseudo bboxes.

Return type

list[Tensor]

get_pseudo_instances(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → Tuple[List[DetDataSample], Optional[dict]]

Get pseudo instances from teacher model.

loss_by_pseudo_instances(batch_inputs: Tensor, batch_data_samples: List[DetDataSample], batch_info: Optional[dict] = None) → dict

Calculate losses from a batch of inputs and pseudo data samples.

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg, which are pseudo_instance or pseudo_panoptic_seg or pseudo_sem_seg in fact.

• batch_info (dict) – Batch information of teacher model forward propagation process. Defaults to None.

Returns

A dictionary of loss components

Return type

dict

rcnn_cls_loss_by_pseudo_instances(x: Tuple[Tensor], unsup_rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample], batch_info: dict) → dict

Calculate classification loss from a batch of inputs and pseudo data samples.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• unsup_rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg, which are pseudo_instance or pseudo_panoptic_seg or pseudo_sem_seg in fact.

• batch_info (dict) – Batch information of teacher model forward propagation process.

Returns

A dictionary of rcnn

classification loss components

Return type

dict[str, Tensor]

rcnn_reg_loss_by_pseudo_instances(x: Tuple[Tensor], unsup_rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → dict

Calculate rcnn regression loss from a batch of inputs and pseudo data samples.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• unsup_rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg, which are pseudo_instance or pseudo_panoptic_seg or pseudo_sem_seg in fact.

Returns

A dictionary of rcnn

regression loss components

Return type

dict[str, Tensor]

rpn_loss_by_pseudo_instances(x: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → dict

Calculate rpn loss from a batch of inputs and pseudo data samples.

Parameters

• x (tuple[Tensor]) – Features from FPN.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg, which are pseudo_instance or pseudo_panoptic_seg or pseudo_sem_seg in fact.

Returns

A dictionary of rpn loss components

Return type

dict

class mmdet.models.detectors.SparseRCNN(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, rpn_head: Optional[Union[ConfigDict, dict]] = None, roi_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Sparse R-CNN: End-to-End Object Detection with Learnable Proposals

class mmdet.models.detectors.TOOD(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of TOOD: Task-aligned One-stage Object Detection.

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• train_cfg (ConfigDict or dict, optional) – The training config of TOOD. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of TOOD. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

class mmdet.models.detectors.TridentFasterRCNN(backbone: Union[ConfigDict, dict], rpn_head: Union[ConfigDict, dict], roi_head: Union[ConfigDict, dict], train_cfg: Union[ConfigDict, dict], test_cfg: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of TridentNet

aug_test(imgs, img_metas, rescale=False)

Test with augmentations.

If rescale is False, then returned bboxes and masks will fit the scale of imgs[0].

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → dict

Copy the batch_data_samples to fit multi-branch.

predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[DetDataSample]

Copy the batch_data_samples to fit multi-branch.

class mmdet.models.detectors.TwoStageDetector(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, rpn_head: Optional[Union[ConfigDict, dict]] = None, roi_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Base class for two-stage detectors.

Two-stage detectors typically consisting of a region proposal network and a task-specific regression head.

extract_feat(batch_inputs: Tensor) → Tuple[Tensor]

Extract features.

Parameters

batch_inputs (Tensor) – Image tensor with shape (N, C, H ,W).

Returns

Multi-level features that may have different resolutions.

Return type

tuple[Tensor]

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → dict

Calculate losses from a batch of inputs and data samples.

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (List[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict

predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[DetDataSample]

Predict results from a batch of inputs and data samples with post- processing.

Parameters

• batch_inputs (Tensor) – Inputs with shape (N, C, H, W).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results. Defaults to True.

Returns

Return the detection results of the input images. The returns value is DetDataSample, which usually contain ‘pred_instances’. And the pred_instances usually contains following keys.

• scores (Tensor): Classification scores, has a shape

  (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape

  (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4),

  the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[DetDataSample]

property with_roi_head: bool

whether the detector has a RoI head

Type

bool

property with_rpn: bool

whether the detector has RPN

Type

bool

class mmdet.models.detectors.TwoStagePanopticSegmentor(backbone: Union[ConfigDict, dict], neck: Optional[Union[ConfigDict, dict]] = None, rpn_head: Optional[Union[ConfigDict, dict]] = None, roi_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, semantic_head: Optional[Union[ConfigDict, dict]] = None, panoptic_fusion_head: Optional[Union[ConfigDict, dict]] = None)

Base class of Two-stage Panoptic Segmentor.

As well as the components in TwoStageDetector, Panoptic Segmentor has extra semantic_head and panoptic_fusion_head.

add_pred_to_datasample(data_samples: List[DetDataSample], results_list: List[PixelData]) → List[DetDataSample]

Add predictions to DetDataSample.

Parameters

• data_samples (list[DetDataSample]) – The annotation data of every samples.

• results_list (List[PixelData]) – Panoptic segmentation results of each image.

Returns

Return the packed panoptic segmentation

results of input images. Each DetDataSample usually contains ‘pred_panoptic_seg’. And the ‘pred_panoptic_seg’ has a key sem_seg, which is a tensor of shape (1, h, w).

Return type

List[DetDataSample]

loss(batch_inputs: Tensor, batch_data_samples: List[DetDataSample]) → dict

Parameters

• batch_inputs (Tensor) – Input images of shape (N, C, H, W). These should usually be mean centered and std scaled.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components.

Return type

dict

predict(batch_inputs: Tensor, batch_data_samples: List[DetDataSample], rescale: bool = True) → List[DetDataSample]

Predict results from a batch of inputs and data samples with post- processing.

Parameters

• batch_inputs (Tensor) – Inputs with shape (N, C, H, W).

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results. Defaults to True.

Returns

Return the packed panoptic segmentation

results of input images. Each DetDataSample usually contains ‘pred_panoptic_seg’. And the ‘pred_panoptic_seg’ has a key sem_seg, which is a tensor of shape (1, h, w).

Return type

List[DetDataSample]

property with_panoptic_fusion_head: bool

whether the detector has panoptic fusion head

Type

bool

property with_semantic_head: bool

whether the detector has semantic head

Type

bool

class mmdet.models.detectors.VFNet(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of `VarifocalNet (VFNet).<https://arxiv.org/abs/2008.13367>`_

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• train_cfg (ConfigDict or dict, optional) – The training config of VFNet. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of VFNet. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

class mmdet.models.detectors.YOLACT(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], mask_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of YOLACT

class mmdet.models.detectors.YOLOF(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of You Only Look One-level Feature

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• train_cfg (ConfigDict or dict, optional) – The training config of YOLOF. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of YOLOF. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Model preprocessing config for processing the input data. it usually includes to_rgb, pad_size_divisor, pad_value, mean and std. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

class mmdet.models.detectors.YOLOV3(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of Yolov3: An incremental improvement

Parameters

• backbone (ConfigDict or dict) – The backbone module.

• neck (ConfigDict or dict) – The neck module.

• bbox_head (ConfigDict or dict) – The bbox head module.

• train_cfg (ConfigDict or dict, optional) – The training config of YOLOX. Default: None.

• test_cfg (ConfigDict or dict, optional) – The testing config of YOLOX. Default: None.

• data_preprocessor (ConfigDict or dict, optional) – Model preprocessing config for processing the input data. it usually includes to_rgb, pad_size_divisor, pad_value, mean and std. Defaults to None.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

class mmdet.models.detectors.YOLOX(backbone: Union[ConfigDict, dict], neck: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict], train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, data_preprocessor: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Implementation of YOLOX: Exceeding YOLO Series in 2021

Parameters

• backbone (ConfigDict or dict) – The backbone config.

• neck (ConfigDict or dict) – The neck config.

• bbox_head (ConfigDict or dict) – The bbox head config.

• train_cfg (ConfigDict or dict, optional) – The training config of YOLOX. Defaults to None.

• test_cfg (ConfigDict or dict, optional) – The testing config of YOLOX. Defaults to None.

• data_preprocessor (ConfigDict or dict, optional) – Config of DetDataPreprocessor to process the input data. Defaults to None.

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Defaults to None.

layers

class mmdet.models.layers.AdaptiveAvgPool2d(output_size: Union[int, None, Tuple[Optional[int], ...]])

Handle empty batch dimension to AdaptiveAvgPool2d.

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.layers.AdaptivePadding(kernel_size=1, stride=1, dilation=1, padding='corner')

Applies padding to input (if needed) so that input can get fully covered by filter you specified. It support two modes “same” and “corner”. The “same” mode is same with “SAME” padding mode in TensorFlow, pad zero around input. The “corner” mode would pad zero to bottom right.

Parameters

• kernel_size (int | tuple) – Size of the kernel:

• stride (int | tuple) – Stride of the filter. Default: 1:

• dilation (int | tuple) – Spacing between kernel elements. Default: 1

• padding (str) – Support “same” and “corner”, “corner” mode would pad zero to bottom right, and “same” mode would pad zero around input. Default: “corner”.

Example

>>> kernel_size = 16
>>> stride = 16
>>> dilation = 1
>>> input = torch.rand(1, 1, 15, 17)
>>> adap_pad = AdaptivePadding(
>>>     kernel_size=kernel_size,
>>>     stride=stride,
>>>     dilation=dilation,
>>>     padding="corner")
>>> out = adap_pad(input)
>>> assert (out.shape[2], out.shape[3]) == (16, 32)
>>> input = torch.rand(1, 1, 16, 17)
>>> out = adap_pad(input)
>>> assert (out.shape[2], out.shape[3]) == (16, 32)

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.layers.CSPLayer(in_channels: int, out_channels: int, expand_ratio: float = 0.5, num_blocks: int = 1, add_identity: bool = True, use_depthwise: bool = False, use_cspnext_block: bool = False, channel_attention: bool = False, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'eps': 0.001, 'momentum': 0.03, 'type': 'BN'}, act_cfg: Union[ConfigDict, dict] = {'type': 'Swish'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Cross Stage Partial Layer.

Parameters

• in_channels (int) – The input channels of the CSP layer.

• out_channels (int) – The output channels of the CSP layer.

• expand_ratio (float) – Ratio to adjust the number of channels of the hidden layer. Defaults to 0.5.

• num_blocks (int) – Number of blocks. Defaults to 1.

• add_identity (bool) – Whether to add identity in blocks. Defaults to True.

• use_cspnext_block (bool) – Whether to use CSPNeXt block. Defaults to False.

• use_depthwise (bool) – Whether to use depthwise separable convolution in blocks. Defaults to False.

• channel_attention (bool) – Whether to add channel attention in each stage. Defaults to True.

• conv_cfg (dict, optional) – Config dict for convolution layer. Defaults to None, which means using conv2d.

• norm_cfg (dict) – Config dict for normalization layer. Defaults to dict(type=’BN’)

• act_cfg (dict) – Config dict for activation layer. Defaults to dict(type=’Swish’)

:param init_cfg (ConfigDict or dict or list[dict] or: list[ConfigDict], optional): Initialization config dict.

Defaults to None.

forward(x: Tensor) → Tensor

Forward function.

class mmdet.models.layers.CdnQueryGenerator(num_classes: int, embed_dims: int, num_matching_queries: int, label_noise_scale: float = 0.5, box_noise_scale: float = 1.0, group_cfg: Optional[Union[ConfigDict, dict]] = None)

Implement query generator of the Contrastive denoising (CDN) proposed in DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

Code is modified from the official github repo.

Parameters

• num_classes (int) – Number of object classes.

• embed_dims (int) – The embedding dimensions of the generated queries.

• num_matching_queries (int) – The queries number of the matching part. Used for generating dn_mask.

• label_noise_scale (float) – The scale of label noise, defaults to 0.5.

• box_noise_scale (float) – The scale of box noise, defaults to 1.0.

• group_cfg (ConfigDict or dict, optional) – The config of the denoising queries grouping, includes dynamic, num_dn_queries, and num_groups. Two grouping strategies, ‘static dn groups’ and ‘dynamic dn groups’, are supported. When dynamic is False, the num_groups should be set, and the number of denoising query groups will always be num_groups. When dynamic is True, the num_dn_queries should be set, and the group number will be dynamic to ensure that the denoising queries number will not exceed num_dn_queries to prevent large fluctuations of memory. Defaults to None.

collate_dn_queries(input_label_query: Tensor, input_bbox_query: Tensor, batch_idx: Tensor, batch_size: int, num_groups: int) → Tuple[Tensor]

Collate generated queries to obtain batched dn queries.

The strategy for query collation is as follow:

        input_queries (num_target_total, query_dim)
P_A1 P_B1 P_B2 N_A1 N_B1 N_B2 P'A1 P'B1 P'B2 N'A1 N'B1 N'B2
  |________ group1 ________|    |________ group2 ________|
                             |
                             V
          P_A1 Pad0 N_A1 Pad0 P'A1 Pad0 N'A1 Pad0
          P_B1 P_B2 N_B1 N_B2 P'B1 P'B2 N'B1 N'B2
           |____ group1 ____| |____ group2 ____|
 batched_queries (batch_size, max_num_target, query_dim)

where query_dim is 4 for bbox and self.embed_dims for label.
Notation: _-group 1; '-group 2;
          A-Sample1(has 1 target); B-sample2(has 2 targets)

Parameters

• input_label_query (Tensor) – The generated label queries of all targets, has shape (num_target_total, embed_dims) where num_target_total = sum(num_target_list).

• input_bbox_query (Tensor) – The generated bbox queries of all targets, has shape (num_target_total, 4) with the last dimension arranged as (cx, cy, w, h).

• batch_idx (Tensor) – The batch index of the corresponding sample for each target, has shape (num_target_total).

• batch_size (int) – The size of the input batch.

• num_groups (int) – The number of denoising query groups.

Returns

Output batched label and bbox queries. - batched_label_query (Tensor): The output batched label queries,

has shape (batch_size, max_num_target, embed_dims).

• batched_bbox_query (Tensor): The output batched bbox queries, has shape (batch_size, max_num_target, 4) with the last dimension arranged as (cx, cy, w, h).

Return type

tuple[Tensor]

generate_dn_bbox_query(gt_bboxes: Tensor, num_groups: int) → Tensor

Generate noisy bboxes and their query embeddings.

The strategy for generating noisy bboxes is as follow:

   +--------------------+
   |      negative      |
   |    +----------+    |
   |    | positive |    |
   |    |    +-----|----+------------+
   |    |    |     |    |            |
   |    +----+-----+    |            |
   |         |          |            |
   +---------+----------+            |
             |                       |
             |        gt bbox        |
             |                       |
             |             +---------+----------+
             |             |         |          |
             |             |    +----+-----+    |
             |             |    |    |     |    |
             +-------------|--- +----+     |    |
                           |    | positive |    |
                           |    +----------+    |
                           |      negative      |
                           +--------------------+

The random noise is added to the top-left and down-right point
positions, hence, normalized (x, y, x, y) format of bboxes are
required. The noisy bboxes of positive queries have the points
both within the inner square, while those of negative queries
have the points both between the inner and outer squares.

Besides, the length of outer square is twice as long as that of the inner square, i.e., self.box_noise_scale * w_or_h / 2. NOTE The noise is added to all the bboxes. Moreover, there is still unconsidered case when one point is within the positive square and the others is between the inner and outer squares.

Parameters

• gt_bboxes (Tensor) – The concatenated gt bboxes of all samples in the batch, has shape (num_target_total, 4) with the last dimension arranged as (cx, cy, w, h) where num_target_total = sum(num_target_list).

• num_groups (int) – The number of denoising query groups.

Returns

The output noisy bboxes, which are embedded by normalized (cx, cy, w, h) format bboxes going through inverse_sigmoid, has shape (num_noisy_targets, 4) with the last dimension arranged as (cx, cy, w, h), where num_noisy_targets = num_target_total * num_groups * 2.

Return type

Tensor

generate_dn_label_query(gt_labels: Tensor, num_groups: int) → Tensor

Generate noisy labels and their query embeddings.

The strategy for generating noisy labels is: Randomly choose labels of self.label_noise_scale * 0.5 proportion and override each of them with a random object category label.

NOTE Not add noise to all labels. Besides, the self.label_noise_scale * 0.5 arg is the ratio of the chosen positions, which is higher than the actual proportion of noisy labels, because the labels to override may be correct. And the gap becomes larger as the number of target categories decreases. The users should notice this and modify the scale arg or the corresponding logic according to specific dataset.

Parameters

• gt_labels (Tensor) – The concatenated gt labels of all samples in the batch, has shape (num_target_total, ) where num_target_total = sum(num_target_list).

• num_groups (int) – The number of denoising query groups.

Returns

The query embeddings of noisy labels, has shape (num_noisy_targets, embed_dims), where num_noisy_targets = num_target_total * num_groups * 2.

Return type

Tensor

generate_dn_mask(max_num_target: int, num_groups: int, device: Union[device, str]) → Tensor

Generate attention mask to prevent information leakage from different denoising groups and matching parts.

               0 0 0 0 1 1 1 1 0 0 0 0 0
               0 0 0 0 1 1 1 1 0 0 0 0 0
               0 0 0 0 1 1 1 1 0 0 0 0 0
               0 0 0 0 1 1 1 1 0 0 0 0 0
               1 1 1 1 0 0 0 0 0 0 0 0 0
               1 1 1 1 0 0 0 0 0 0 0 0 0
               1 1 1 1 0 0 0 0 0 0 0 0 0
               1 1 1 1 0 0 0 0 0 0 0 0 0
               1 1 1 1 1 1 1 1 0 0 0 0 0
               1 1 1 1 1 1 1 1 0 0 0 0 0
               1 1 1 1 1 1 1 1 0 0 0 0 0
               1 1 1 1 1 1 1 1 0 0 0 0 0
               1 1 1 1 1 1 1 1 0 0 0 0 0
max_num_target |_|           |_________| num_matching_queries
               |_____________| num_denoising_queries

      1 -> True  (Masked), means 'can not see'.
      0 -> False (UnMasked), means 'can see'.

Parameters

• max_num_target (int) – The max target number of the input batch samples.

• num_groups (int) – The number of denoising query groups.

• (obj (device) – device or str): The device of generated mask.

Returns

The attention mask to prevent information leakage from different denoising groups and matching parts, will be used as self_attn_mask of the decoder, has shape (num_queries_total, num_queries_total), where num_queries_total is the sum of num_denoising_queries and num_matching_queries.

Return type

Tensor

get_num_groups(max_num_target: Optional[int] = None) → int

Calculate denoising query groups number.

Two grouping strategies, ‘static dn groups’ and ‘dynamic dn groups’, are supported. When self.dynamic_dn_groups is False, the number of denoising query groups will always be self.num_groups. When self.dynamic_dn_groups is True, the group number will be dynamic, ensuring the denoising queries number will not exceed self.num_dn_queries to prevent large fluctuations of memory.

NOTE The num_group is shared for different samples in a batch. When the target numbers in the samples varies, the denoising queries of the samples containing fewer targets are padded to the max length.

Parameters

max_num_target (int, optional) – The max target number of the batch samples. It will only be used when self.dynamic_dn_groups is True. Defaults to None.

Returns

The denoising group number of the current batch.

Return type

int

class mmdet.models.layers.ChannelAttention(channels: int, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Channel attention Module.

Parameters

• channels (int) – The input (and output) channels of the attention layer.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Defaults to None

forward(x: Tensor) → Tensor

Forward function for ChannelAttention.

class mmdet.models.layers.ConditionalAttention(embed_dims: int, num_heads: int, attn_drop: float = 0.0, proj_drop: float = 0.0, cross_attn: bool = False, keep_query_pos: bool = False, batch_first: bool = True, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

A wrapper of conditional attention, dropout and residual connection.

Parameters

• embed_dims (int) – The embedding dimension.

• num_heads (int) – Parallel attention heads.

• attn_drop (float) – A Dropout layer on attn_output_weights. Default: 0.0.

• proj_drop – A Dropout layer after nn.MultiheadAttention. Default: 0.0.

• cross_attn (bool) – Whether the attention module is for cross attention. Default: False

• keep_query_pos (bool) – Whether to transform query_pos before cross attention. Default: False.

• batch_first (bool) –

  When it is True, Key, Query and Value are shape of (batch, n, embed_dim), otherwise (n, batch, embed_dim).

  Default: True.

• (obj (init_cfg) – mmcv.ConfigDict): The Config for initialization. Default: None.

forward(query: Tensor, key: Tensor, query_pos: Optional[Tensor] = None, ref_sine_embed: Optional[Tensor] = None, key_pos: Optional[Tensor] = None, attn_mask: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None, is_first: bool = False) → Tensor

Forward function for ConditionalAttention.

Parameters

• query (Tensor) – The input query with shape [bs, num_queries, embed_dims].

• key (Tensor) – The key tensor with shape [bs, num_keys, embed_dims]. If None, the query will be used. Defaults to None.

• query_pos (Tensor) – The positional encoding for query in self attention, with the same shape as x. If not None, it will be added to x before forward function. Defaults to None.

• query_sine_embed (Tensor) – The positional encoding for query in cross attention, with the same shape as x. If not None, it will be added to x before forward function. Defaults to None.

• key_pos (Tensor) – The positional encoding for key, with the same shape as key. Defaults to None. If not None, it will be added to key before forward function. If None, and query_pos has the same shape as key, then query_pos will be used for key_pos. Defaults to None.

• attn_mask (Tensor) – ByteTensor mask with shape [num_queries, num_keys]. Same in nn.MultiheadAttention.forward. Defaults to None.

• key_padding_mask (Tensor) – ByteTensor with shape [bs, num_keys]. Defaults to None.

• is_first (bool) – A indicator to tell whether the current layer is the first layer of the decoder. Defaults to False.

Returns

forwarded results with shape [bs, num_queries, embed_dims].

Return type

Tensor

forward_attn(query: Tensor, key: Tensor, value: Tensor, attn_mask: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None) → Tuple[Tensor]

Forward process for ConditionalAttention.

Parameters

• query (Tensor) – The input query with shape [bs, num_queries, embed_dims].

• key (Tensor) – The key tensor with shape [bs, num_keys, embed_dims]. If None, the query will be used. Defaults to None.

• value (Tensor) – The value tensor with same shape as key. Same in nn.MultiheadAttention.forward. Defaults to None. If None, the key will be used.

• attn_mask (Tensor) – ByteTensor mask with shape [num_queries, num_keys]. Same in nn.MultiheadAttention.forward. Defaults to None.

• key_padding_mask (Tensor) – ByteTensor with shape [bs, num_keys]. Defaults to None.

Returns

Attention outputs of shape \((N, L, E)\), where \(N\) is the batch size, \(L\) is the target sequence length , and \(E\) is the embedding dimension embed_dim. Attention weights per head of shape :math:` (num_heads, L, S)`. where \(N\) is batch size, \(L\) is target sequence length, and \(S\) is the source sequence length.

Return type

Tuple[Tensor]

class mmdet.models.layers.ConditionalDetrTransformerDecoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], post_norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, return_intermediate: bool = True, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Decoder of Conditional DETR.

forward(query: Tensor, key: Optional[Tensor] = None, query_pos: Optional[Tensor] = None, key_pos: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None)

Forward function of decoder.

Parameters

• query (Tensor) – The input query with shape (bs, num_queries, dim).

• key (Tensor) – The input key with shape (bs, num_keys, dim) If None, the query will be used. Defaults to None.

• query_pos (Tensor) – The positional encoding for query, with the same shape as query. If not None, it will be added to query before forward function. Defaults to None.

• key_pos (Tensor) – The positional encoding for key, with the same shape as key. If not None, it will be added to key before forward function. If None, and query_pos has the same shape as key, then query_pos will be used as key_pos. Defaults to None.

• key_padding_mask (Tensor) – ByteTensor with shape (bs, num_keys). Defaults to None.

Returns

forwarded results with shape (num_decoder_layers, bs, num_queries, dim) if return_intermediate is True, otherwise with shape (1, bs, num_queries, dim). References with shape (bs, num_queries, 2).

Return type

List[Tensor]

class mmdet.models.layers.ConditionalDetrTransformerDecoderLayer(self_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, cross_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, ffn_cfg: Optional[Union[ConfigDict, dict]] = {'act_cfg': {'inplace': True, 'type': 'ReLU'}, 'embed_dims': 256, 'feedforward_channels': 1024, 'ffn_drop': 0.0, 'num_fcs': 2}, norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Implements decoder layer in Conditional DETR transformer.

forward(query: Tensor, key: Optional[Tensor] = None, query_pos: Optional[Tensor] = None, key_pos: Optional[Tensor] = None, self_attn_masks: Optional[Tensor] = None, cross_attn_masks: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None, ref_sine_embed: Optional[Tensor] = None, is_first: bool = False)

Parameters

• query (Tensor) – The input query, has shape (bs, num_queries, dim)

• key (Tensor, optional) – The input key, has shape (bs, num_keys, dim). If None, the query will be used. Defaults to None.

• query_pos (Tensor, optional) – The positional encoding for query, has the same shape as query. If not None, it will be added to query before forward function. Defaults to None.

• ref_sine_embed (Tensor) – The positional encoding for query in cross attention, with the same shape as x. Defaults to None.

• key_pos (Tensor, optional) – The positional encoding for key, has the same shape as key. If not None, it will be added to key before forward function. If None, and query_pos has the same shape as key, then query_pos will be used for key_pos. Defaults to None.

• self_attn_masks (Tensor, optional) – ByteTensor mask, has shape (num_queries, num_keys), Same in nn.MultiheadAttention. forward. Defaults to None.

• cross_attn_masks (Tensor, optional) – ByteTensor mask, has shape (num_queries, num_keys), Same in nn.MultiheadAttention. forward. Defaults to None.

• key_padding_mask (Tensor, optional) – ByteTensor, has shape (bs, num_keys). Defaults to None.

• is_first (bool) – A indicator to tell whether the current layer is the first layer of the decoder. Defaults to False.

Returns

Forwarded results, has shape (bs, num_queries, dim).

Return type

Tensor

class mmdet.models.layers.ConvUpsample(in_channels, inner_channels, num_layers=1, num_upsample=None, conv_cfg=None, norm_cfg=None, init_cfg=None, **kwargs)

ConvUpsample performs 2x upsampling after Conv.

There are several ConvModule layers. In the first few layers, upsampling will be applied after each layer of convolution. The number of upsampling must be no more than the number of ConvModule layers.

Parameters

• in_channels (int) – Number of channels in the input feature map.

• inner_channels (int) – Number of channels produced by the convolution.

• num_layers (int) – Number of convolution layers.

• num_upsample (int | optional) – Number of upsampling layer. Must be no more than num_layers. Upsampling will be applied after the first num_upsample layers of convolution. Default: num_layers.

• conv_cfg (dict) – Config dict for convolution layer. Default: None, which means using conv2d.

• norm_cfg (dict) – Config dict for normalization layer. Default: None.

• init_cfg (dict) – Config dict for initialization. Default: None.

• kwargs (key word augments) – Other augments used in ConvModule.

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.layers.DABDetrTransformerDecoder(*args, query_dim: int = 4, query_scale_type: str = 'cond_elewise', with_modulated_hw_attn: bool = True, **kwargs)

Decoder of DAB-DETR.

Parameters

• query_dim (int) – The last dimension of query pos, 4 for anchor format, 2 for point format. Defaults to 4.

• query_scale_type (str) – Type of transformation applied to content query. Defaults to cond_elewise.

• with_modulated_hw_attn (bool) – Whether to inject h&w info during cross conditional attention. Defaults to True.

forward(query: Tensor, key: Tensor, query_pos: Tensor, key_pos: Tensor, reg_branches: Module, key_padding_mask: Optional[Tensor] = None, **kwargs) → List[Tensor]

Forward function of decoder.

Parameters

• query (Tensor) – The input query with shape (bs, num_queries, dim).

• key (Tensor) – The input key with shape (bs, num_keys, dim).

• query_pos (Tensor) – The positional encoding for query, with the same shape as query.

• key_pos (Tensor) – The positional encoding for key, with the same shape as key.

• reg_branches (nn.Module) – The regression branch for dynamically updating references in each layer.

• key_padding_mask (Tensor) – ByteTensor with shape (bs, num_keys). Defaults to None.

Returns

forwarded results with shape (num_decoder_layers, bs, num_queries, dim) if return_intermediate is True, otherwise with shape (1, bs, num_queries, dim). references with shape (num_decoder_layers, bs, num_queries, 2/4).

Return type

List[Tensor]

class mmdet.models.layers.DABDetrTransformerDecoderLayer(self_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, cross_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, ffn_cfg: Optional[Union[ConfigDict, dict]] = {'act_cfg': {'inplace': True, 'type': 'ReLU'}, 'embed_dims': 256, 'feedforward_channels': 1024, 'ffn_drop': 0.0, 'num_fcs': 2}, norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Implements decoder layer in DAB-DETR transformer.

forward(query: Tensor, key: Tensor, query_pos: Tensor, key_pos: Tensor, ref_sine_embed: Optional[Tensor] = None, self_attn_masks: Optional[Tensor] = None, cross_attn_masks: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None, is_first: bool = False, **kwargs) → Tensor

Parameters

• query (Tensor) – The input query with shape [bs, num_queries, dim].

• key (Tensor) – The key tensor with shape [bs, num_keys, dim].

• query_pos (Tensor) – The positional encoding for query in self attention, with the same shape as x.

• key_pos (Tensor) – The positional encoding for key, with the same shape as key.

• ref_sine_embed (Tensor) – The positional encoding for query in cross attention, with the same shape as x. Defaults to None.

• self_attn_masks (Tensor) – ByteTensor mask with shape [num_queries, num_keys]. Same in nn.MultiheadAttention.forward. Defaults to None.

• cross_attn_masks (Tensor) – ByteTensor mask with shape [num_queries, num_keys]. Same in nn.MultiheadAttention.forward. Defaults to None.

• key_padding_mask (Tensor) – ByteTensor with shape [bs, num_keys]. Defaults to None.

• is_first (bool) – A indicator to tell whether the current layer is the first layer of the decoder. Defaults to False.

Returns

forwarded results with shape [bs, num_queries, dim].

Return type

Tensor

class mmdet.models.layers.DABDetrTransformerEncoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], num_cp: int = -1, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Encoder of DAB-DETR.

forward(query: Tensor, query_pos: Tensor, key_padding_mask: Tensor, **kwargs)

Forward function of encoder.

Parameters

• query (Tensor) – Input queries of encoder, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional embeddings of the queries, has shape (bs, num_feat_points, dim).

• key_padding_mask (Tensor) – ByteTensor, the key padding mask of the queries, has shape (bs, num_feat_points).

Returns

With shape (num_queries, bs, dim).

Return type

Tensor

class mmdet.models.layers.DDQTransformerDecoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], post_norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, return_intermediate: bool = True, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Transformer decoder of DDQ.

forward(query: Tensor, value: Tensor, key_padding_mask: Tensor, self_attn_mask: Tensor, reference_points: Tensor, spatial_shapes: Tensor, level_start_index: Tensor, valid_ratios: Tensor, reg_branches: ModuleList, **kwargs) → Tensor

Forward function of Transformer decoder.

Parameters

• query (Tensor) – The input query, has shape (bs, num_queries, dims).

• value (Tensor) – The input values, has shape (bs, num_value, dim).

• key_padding_mask (Tensor) – The key_padding_mask of cross_attn input. ByteTensor, has shape (bs, num_value).

• self_attn_mask (Tensor) – The attention mask to prevent information leakage from different denoising groups, distinct queries and dense queries, has shape (num_queries_total, num_queries_total). It will be updated for distinct queries selection in this forward function. It is None when self.training is False.

• reference_points (Tensor) – The initial reference, has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• level_start_index (Tensor) – The start index of each level. A tensor has shape (num_levels, ) and can be represented as [0, h_0*w_0, h_0*w_0+h_1*w_1, …].

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

• reg_branches – (obj:nn.ModuleList): Used for refining the regression results.

Returns

Output queries and references of Transformer

decoder

• query (Tensor): Output embeddings of the last decoder, has shape (bs, num_queries, embed_dims) when return_intermediate is False. Otherwise, Intermediate output embeddings of all decoder layers, has shape (num_decoder_layers, bs, num_queries, embed_dims).

• reference_points (Tensor): The reference of the last decoder layer, has shape (bs, num_queries, 4) when return_intermediate is False. Otherwise, Intermediate references of all decoder layers, has shape (1 + num_decoder_layers, bs, num_queries, 4). The coordinates are arranged as (cx, cy, w, h).

Return type

tuple[Tensor]

select_distinct_queries(reference_points: Tensor, query: Tensor, self_attn_mask: Tensor, layer_index)

Get updated self_attn_mask for distinct queries selection, it is used in self attention layers of decoder.

Parameters

• reference_points (Tensor) – The input reference of decoder, has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• query (Tensor) – The input query of decoder, has shape (bs, num_queries, dims).

• self_attn_mask (Tensor) – The input self attention mask of last decoder layer, has shape (bs, num_queries_total, num_queries_total).

• layer_index (int) – Last decoder layer index, used to get classification score of last layer output, for distinct queries selection.

Returns

self_attn_mask used in self attention layers

of decoder, has shape (bs, num_queries_total, num_queries_total).

Return type

Tensor

class mmdet.models.layers.DeformableDetrTransformerDecoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], post_norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, return_intermediate: bool = True, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Transformer Decoder of Deformable DETR.

forward(query: Tensor, query_pos: Tensor, value: Tensor, key_padding_mask: Tensor, reference_points: Tensor, spatial_shapes: Tensor, level_start_index: Tensor, valid_ratios: Tensor, reg_branches: Optional[Module] = None, **kwargs) → Tuple[Tensor]

Forward function of Transformer decoder.

Parameters

• query (Tensor) – The input queries, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The input positional query, has shape (bs, num_queries, dim). It will be added to query before forward function.

• value (Tensor) – The input values, has shape (bs, num_value, dim).

• key_padding_mask (Tensor) – The key_padding_mask of cross_attn input. ByteTensor, has shape (bs, num_value).

• reference_points (Tensor) – The initial reference, has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h) when as_two_stage is True, otherwise has shape (bs, num_queries, 2) with the last dimension arranged as (cx, cy).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• level_start_index (Tensor) – The start index of each level. A tensor has shape (num_levels, ) and can be represented as [0, h_0*w_0, h_0*w_0+h_1*w_1, …].

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

• reg_branches – (obj:nn.ModuleList, optional): Used for refining the regression results. Only would be passed when with_box_refine is True, otherwise would be None.

Returns

Outputs of Deformable Transformer Decoder.

• output (Tensor): Output embeddings of the last decoder, has shape (num_queries, bs, embed_dims) when return_intermediate is False. Otherwise, Intermediate output embeddings of all decoder layers, has shape (num_decoder_layers, num_queries, bs, embed_dims).

• reference_points (Tensor): The reference of the last decoder layer, has shape (bs, num_queries, 4) when return_intermediate is False. Otherwise, Intermediate references of all decoder layers, has shape (num_decoder_layers, bs, num_queries, 4). The coordinates are arranged as (cx, cy, w, h)

Return type

tuple[Tensor]

class mmdet.models.layers.DeformableDetrTransformerDecoderLayer(self_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, cross_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, ffn_cfg: Optional[Union[ConfigDict, dict]] = {'act_cfg': {'inplace': True, 'type': 'ReLU'}, 'embed_dims': 256, 'feedforward_channels': 1024, 'ffn_drop': 0.0, 'num_fcs': 2}, norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Decoder layer of Deformable DETR.

class mmdet.models.layers.DeformableDetrTransformerEncoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], num_cp: int = -1, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Transformer encoder of Deformable DETR.

forward(query: Tensor, query_pos: Tensor, key_padding_mask: Tensor, spatial_shapes: Tensor, level_start_index: Tensor, valid_ratios: Tensor, **kwargs) → Tensor

Forward function of Transformer encoder.

Parameters

• query (Tensor) – The input query, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional encoding for query, has shape (bs, num_queries, dim).

• key_padding_mask (Tensor) – The key_padding_mask of self_attn input. ByteTensor, has shape (bs, num_queries).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• level_start_index (Tensor) – The start index of each level. A tensor has shape (num_levels, ) and can be represented as [0, h_0*w_0, h_0*w_0+h_1*w_1, …].

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

Returns

Output queries of Transformer encoder, which is also called ‘encoder output embeddings’ or ‘memory’, has shape (bs, num_queries, dim)

Return type

Tensor

static get_encoder_reference_points(spatial_shapes: Tensor, valid_ratios: Tensor, device: Union[device, str]) → Tensor

Get the reference points used in encoder.

Parameters

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

• (obj (device) – device or str): The device acquired by the reference_points.

Returns

Reference points used in decoder, has shape (bs, length, num_levels, 2).

Return type

Tensor

class mmdet.models.layers.DeformableDetrTransformerEncoderLayer(self_attn_cfg: Optional[Union[ConfigDict, dict]] = {'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, ffn_cfg: Optional[Union[ConfigDict, dict]] = {'act_cfg': {'inplace': True, 'type': 'ReLU'}, 'embed_dims': 256, 'feedforward_channels': 1024, 'ffn_drop': 0.0, 'num_fcs': 2}, norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Encoder layer of Deformable DETR.

class mmdet.models.layers.DetrTransformerDecoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], post_norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, return_intermediate: bool = True, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Decoder of DETR.

Parameters

• num_layers (int) – Number of decoder layers.

• layer_cfg (ConfigDict or dict) – the config of each encoder layer. All the layers will share the same config.

• post_norm_cfg (ConfigDict or dict, optional) – Config of the post normalization layer. Defaults to LN.

• return_intermediate (bool, optional) – Whether to return outputs of intermediate layers. Defaults to True,

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

forward(query: Tensor, key: Tensor, value: Tensor, query_pos: Tensor, key_pos: Tensor, key_padding_mask: Tensor, **kwargs) → Tensor

Forward function of decoder :param query: The input query, has shape (bs, num_queries, dim). :type query: Tensor :param key: The input key, has shape (bs, num_keys, dim). :type key: Tensor :param value: The input value with the same shape as key. :type value: Tensor :param query_pos: The positional encoding for query, with the

same shape as query.

Parameters

• key_pos (Tensor) – The positional encoding for key, with the same shape as key.

• key_padding_mask (Tensor) – The key_padding_mask of cross_attn input. ByteTensor, has shape (bs, num_value).

Returns

The forwarded results will have shape (num_decoder_layers, bs, num_queries, dim) if return_intermediate is True else (1, bs, num_queries, dim).

Return type

Tensor

class mmdet.models.layers.DetrTransformerDecoderLayer(self_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, cross_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, ffn_cfg: Optional[Union[ConfigDict, dict]] = {'act_cfg': {'inplace': True, 'type': 'ReLU'}, 'embed_dims': 256, 'feedforward_channels': 1024, 'ffn_drop': 0.0, 'num_fcs': 2}, norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Implements decoder layer in DETR transformer.

Parameters

• self_attn_cfg (ConfigDict or dict, optional) – Config for self attention.

• cross_attn_cfg (ConfigDict or dict, optional) – Config for cross attention.

• ffn_cfg (ConfigDict or dict, optional) – Config for FFN.

• norm_cfg (ConfigDict or dict, optional) – Config for normalization layers. All the layers will share the same config. Defaults to LN.

• init_cfg (ConfigDict or dict, optional) – Config to control the initialization. Defaults to None.

forward(query: Tensor, key: Optional[Tensor] = None, value: Optional[Tensor] = None, query_pos: Optional[Tensor] = None, key_pos: Optional[Tensor] = None, self_attn_mask: Optional[Tensor] = None, cross_attn_mask: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None, **kwargs) → Tensor

Parameters

• query (Tensor) – The input query, has shape (bs, num_queries, dim).

• key (Tensor, optional) – The input key, has shape (bs, num_keys, dim). If None, the query will be used. Defaults to None.

• value (Tensor, optional) – The input value, has the same shape as key, as in nn.MultiheadAttention.forward. If None, the key will be used. Defaults to None.

• query_pos (Tensor, optional) – The positional encoding for query, has the same shape as query. If not None, it will be added to query before forward function. Defaults to None.

• key_pos (Tensor, optional) – The positional encoding for key, has the same shape as key. If not None, it will be added to key before forward function. If None, and query_pos has the same shape as key, then query_pos will be used for key_pos. Defaults to None.

• self_attn_mask (Tensor, optional) – ByteTensor mask, has shape (num_queries, num_keys), as in nn.MultiheadAttention.forward. Defaults to None.

• cross_attn_mask (Tensor, optional) – ByteTensor mask, has shape (num_queries, num_keys), as in nn.MultiheadAttention.forward. Defaults to None.

• key_padding_mask (Tensor, optional) – The key_padding_mask of self_attn input. ByteTensor, has shape (bs, num_value). Defaults to None.

Returns

forwarded results, has shape (bs, num_queries, dim).

Return type

Tensor

class mmdet.models.layers.DetrTransformerEncoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], num_cp: int = -1, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Encoder of DETR.

Parameters

• num_layers (int) – Number of encoder layers.

• layer_cfg (ConfigDict or dict) – the config of each encoder layer. All the layers will share the same config.

• num_cp (int) – Number of checkpointing blocks in encoder layer. Default to -1.

• init_cfg (ConfigDict or dict, optional) – the config to control the initialization. Defaults to None.

forward(query: Tensor, query_pos: Tensor, key_padding_mask: Tensor, **kwargs) → Tensor

Forward function of encoder.

Parameters

• query (Tensor) – Input queries of encoder, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional embeddings of the queries, has shape (bs, num_queries, dim).

• key_padding_mask (Tensor) – The key_padding_mask of self_attn input. ByteTensor, has shape (bs, num_queries).

Returns

Has shape (bs, num_queries, dim) if batch_first is True, otherwise (num_queries, bs, dim).

Return type

Tensor

class mmdet.models.layers.DetrTransformerEncoderLayer(self_attn_cfg: Optional[Union[ConfigDict, dict]] = {'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, ffn_cfg: Optional[Union[ConfigDict, dict]] = {'act_cfg': {'inplace': True, 'type': 'ReLU'}, 'embed_dims': 256, 'feedforward_channels': 1024, 'ffn_drop': 0.0, 'num_fcs': 2}, norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Implements encoder layer in DETR transformer.

Parameters

• self_attn_cfg (ConfigDict or dict, optional) – Config for self attention.

• ffn_cfg (ConfigDict or dict, optional) – Config for FFN.

• norm_cfg (ConfigDict or dict, optional) – Config for normalization layers. All the layers will share the same config. Defaults to LN.

• init_cfg (ConfigDict or dict, optional) – Config to control the initialization. Defaults to None.

forward(query: Tensor, query_pos: Tensor, key_padding_mask: Tensor, **kwargs) → Tensor

Forward function of an encoder layer.

Parameters

• query (Tensor) – The input query, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional encoding for query, with the same shape as query.

• key_padding_mask (Tensor) – The key_padding_mask of self_attn input. ByteTensor. has shape (bs, num_queries).

Returns

forwarded results, has shape (bs, num_queries, dim).

Return type

Tensor

class mmdet.models.layers.DinoTransformerDecoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], post_norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, return_intermediate: bool = True, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Transformer decoder of DINO.

forward(query: Tensor, value: Tensor, key_padding_mask: Tensor, self_attn_mask: Tensor, reference_points: Tensor, spatial_shapes: Tensor, level_start_index: Tensor, valid_ratios: Tensor, reg_branches: ModuleList, **kwargs) → Tuple[Tensor]

Forward function of Transformer decoder.

Parameters

• query (Tensor) – The input query, has shape (num_queries, bs, dim).

• value (Tensor) – The input values, has shape (num_value, bs, dim).

• key_padding_mask (Tensor) – The key_padding_mask of self_attn input. ByteTensor, has shape (num_queries, bs).

• self_attn_mask (Tensor) – The attention mask to prevent information leakage from different denoising groups and matching parts, has shape (num_queries_total, num_queries_total). It is None when self.training is False.

• reference_points (Tensor) – The initial reference, has shape (bs, num_queries, 4) with the last dimension arranged as (cx, cy, w, h).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• level_start_index (Tensor) – The start index of each level. A tensor has shape (num_levels, ) and can be represented as [0, h_0*w_0, h_0*w_0+h_1*w_1, …].

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

• reg_branches – (obj:nn.ModuleList): Used for refining the regression results.

Returns

Output queries and references of Transformer

decoder

• query (Tensor): Output embeddings of the last decoder, has shape (num_queries, bs, embed_dims) when return_intermediate is False. Otherwise, Intermediate output embeddings of all decoder layers, has shape (num_decoder_layers, num_queries, bs, embed_dims).

• reference_points (Tensor): The reference of the last decoder layer, has shape (bs, num_queries, 4) when return_intermediate is False. Otherwise, Intermediate references of all decoder layers, has shape (num_decoder_layers, bs, num_queries, 4). The coordinates are arranged as (cx, cy, w, h)

Return type

tuple[Tensor]

class mmdet.models.layers.DropBlock(drop_prob, block_size, warmup_iters=2000, **kwargs)

Randomly drop some regions of feature maps.

Please refer to the method proposed in DropBlock for details.

Parameters

• drop_prob (float) – The probability of dropping each block.

• block_size (int) – The size of dropped blocks.

• warmup_iters (int) – The drop probability will linearly increase from 0 to drop_prob during the first warmup_iters iterations. Default: 2000.

extra_repr()

Set the extra representation of the module.

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

forward(x)

Parameters

x (Tensor) – Input feature map on which some areas will be randomly dropped.

Returns

The tensor after DropBlock layer.

Return type

Tensor

class mmdet.models.layers.DyReLU(channels: int, ratio: int = 4, conv_cfg: Optional[Union[ConfigDict, dict]] = None, act_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = ({'type': 'ReLU'}, {'type': 'HSigmoid', 'bias': 3.0, 'divisor': 6.0}), init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Dynamic ReLU (DyReLU) module.

See Dynamic ReLU for details. Current implementation is specialized for task-aware attention in DyHead. HSigmoid arguments in default act_cfg follow DyHead official code. https://github.com/microsoft/DynamicHead/blob/master/dyhead/dyrelu.py

Parameters

• channels (int) – The input (and output) channels of DyReLU module.

• ratio (int) – Squeeze ratio in Squeeze-and-Excitation-like module, the intermediate channel will be int(channels/ratio). Defaults to 4.

• conv_cfg (None or dict) – Config dict for convolution layer. Defaults to None, which means using conv2d.

• act_cfg (dict or Sequence[dict]) – Config dict for activation layer. If act_cfg is a dict, two activation layers will be configured by this dict. If act_cfg is a sequence of dicts, the first activation layer will be configured by the first dict and the second activation layer will be configured by the second dict. Defaults to (dict(type=’ReLU’), dict(type=’HSigmoid’, bias=3.0, divisor=6.0))

• init_cfg (dict or list[dict], optional) – Initialization config dict. Defaults to None

forward(x: Tensor) → Tensor

Forward function.

class mmdet.models.layers.DynamicConv(in_channels: int = 256, feat_channels: int = 64, out_channels: Optional[int] = None, input_feat_shape: int = 7, with_proj: bool = True, act_cfg: Optional[Union[ConfigDict, dict]] = {'inplace': True, 'type': 'ReLU'}, norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Implements Dynamic Convolution.

This module generate parameters for each sample and use bmm to implement 1*1 convolution. Code is modified from the official github repo .

Parameters

• in_channels (int) – The input feature channel. Defaults to 256.

• feat_channels (int) – The inner feature channel. Defaults to 64.

• out_channels (int, optional) – The output feature channel. When not specified, it will be set to in_channels by default

• input_feat_shape (int) – The shape of input feature. Defaults to 7.

• with_proj (bool) – Project two-dimentional feature to one-dimentional feature. Default to True.

• act_cfg (dict) – The activation config for DynamicConv.

• norm_cfg (dict) – Config dict for normalization layer. Default layer normalization.

• (obj (init_cfg) – mmengine.ConfigDict): The Config for initialization. Default: None.

forward(param_feature: Tensor, input_feature: Tensor) → Tensor

Forward function for DynamicConv.

Parameters

• param_feature (Tensor) – The feature can be used to generate the parameter, has shape (num_all_proposals, in_channels).

• input_feature (Tensor) – Feature that interact with parameters, has shape (num_all_proposals, in_channels, H, W).

Returns

The output feature has shape (num_all_proposals, out_channels).

Return type

Tensor

class mmdet.models.layers.ExpMomentumEMA(model: Module, momentum: float = 0.0002, gamma: int = 2000, interval=1, device: Optional[device] = None, update_buffers: bool = False)

Exponential moving average (EMA) with exponential momentum strategy, which is used in YOLOX.

Parameters

• model (nn.Module) – The model to be averaged.

• momentum (float) –

  The momentum used for updating ema parameter.

  Ema’s parameter are updated with the formula:

  averaged_param = (1-momentum) * averaged_param + momentum * source_param. Defaults to 0.0002.

• gamma (int) – Use a larger momentum early in training and gradually annealing to a smaller value to update the ema model smoothly. The momentum is calculated as (1 - momentum) * exp(-(1 + steps) / gamma) + momentum. Defaults to 2000.

• interval (int) – Interval between two updates. Defaults to 1.

• device (torch.device, optional) – If provided, the averaged model will be stored on the device. Defaults to None.

• update_buffers (bool) – if True, it will compute running averages for both the parameters and the buffers of the model. Defaults to False.

avg_func(averaged_param: Tensor, source_param: Tensor, steps: int) → None

Compute the moving average of the parameters using the exponential momentum strategy.

Parameters

• averaged_param (Tensor) – The averaged parameters.

• source_param (Tensor) – The source parameters.

• steps (int) – The number of times the parameters have been updated.

class mmdet.models.layers.FrozenBatchNorm2d(num_features, eps=1e-05, **kwargs)

BatchNorm2d where the batch statistics and the affine parameters are fixed.

It contains non-trainable buffers called “weight” and “bias”, “running_mean”, “running_var”, initialized to perform identity transformation. :param num_features: \(C\) from an expected input of size

\((N, C, H, W)\).

Parameters

eps (float) – a value added to the denominator for numerical stability. Default: 1e-5

classmethod convert_frozen_batchnorm(module)

Convert all BatchNorm/SyncBatchNorm in module into FrozenBatchNorm.

Parameters

module (torch.nn.Module) –

Returns

If module is BatchNorm/SyncBatchNorm, returns a new module. Otherwise, in-place convert module and return it.

Similar to convert_sync_batchnorm in https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.layers.InvertedResidual(in_channels, out_channels, mid_channels, kernel_size=3, stride=1, se_cfg=None, with_expand_conv=True, conv_cfg=None, norm_cfg={'type': 'BN'}, act_cfg={'type': 'ReLU'}, drop_path_rate=0.0, with_cp=False, init_cfg=None)

Inverted Residual Block.

Parameters

• in_channels (int) – The input channels of this Module.

• out_channels (int) – The output channels of this Module.

• mid_channels (int) – The input channels of the depthwise convolution.

• kernel_size (int) – The kernel size of the depthwise convolution. Default: 3.

• stride (int) – The stride of the depthwise convolution. Default: 1.

• se_cfg (dict) – Config dict for se layer. Default: None, which means no se layer.

• with_expand_conv (bool) – Use expand conv or not. If set False, mid_channels must be the same with in_channels. Default: True.

• conv_cfg (dict) – Config dict for convolution layer. Default: None, which means using conv2d.

• norm_cfg (dict) – Config dict for normalization layer. Default: dict(type=’BN’).

• act_cfg (dict) – Config dict for activation layer. Default: dict(type=’ReLU’).

• drop_path_rate (float) – stochastic depth rate. Defaults to 0.

• with_cp (bool) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed. Default: False.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

Returns

The output tensor.

Return type

Tensor

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.layers.LearnedPositionalEncoding(num_feats: int, row_num_embed: int = 50, col_num_embed: int = 50, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Embedding', 'type': 'Uniform'})

Position embedding with learnable embedding weights.

Parameters

• num_feats (int) – The feature dimension for each position along x-axis or y-axis. The final returned dimension for each position is 2 times of this value.

• row_num_embed (int, optional) – The dictionary size of row embeddings. Defaults to 50.

• col_num_embed (int, optional) – The dictionary size of col embeddings. Defaults to 50.

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(mask: Tensor) → Tensor

Forward function for LearnedPositionalEncoding.

Parameters

mask (Tensor) – ByteTensor mask. Non-zero values representing ignored positions, while zero values means valid positions for this image. Shape [bs, h, w].

Returns

Returned position embedding with shape

[bs, num_feats*2, h, w].

Return type

pos (Tensor)

class mmdet.models.layers.MLP(input_dim: int, hidden_dim: int, output_dim: int, num_layers: int)

Very simple multi-layer perceptron (also called FFN) with relu. Mostly used in DETR series detectors.

Parameters

• input_dim (int) – Feature dim of the input tensor.

• hidden_dim (int) – Feature dim of the hidden layer.

• output_dim (int) – Feature dim of the output tensor.

• num_layers (int) – Number of FFN layers. As the last layer of MLP only contains FFN (Linear).

forward(x: Tensor) → Tensor

Forward function of MLP.

Parameters

x (Tensor) – The input feature, has shape (num_queries, bs, input_dim).

Returns

The output feature, has shape

(num_queries, bs, output_dim).

Return type

Tensor

class mmdet.models.layers.MSDeformAttnPixelDecoder(in_channels: Union[List[int], Tuple[int]] = [256, 512, 1024, 2048], strides: Union[List[int], Tuple[int]] = [4, 8, 16, 32], feat_channels: int = 256, out_channels: int = 256, num_outs: int = 3, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 32, 'type': 'GN'}, act_cfg: Union[ConfigDict, dict] = {'type': 'ReLU'}, encoder: Optional[Union[ConfigDict, dict]] = None, positional_encoding: Union[ConfigDict, dict] = {'normalize': True, 'num_feats': 128}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Pixel decoder with multi-scale deformable attention.

Parameters

• in_channels (list[int] | tuple[int]) – Number of channels in the input feature maps.

• strides (list[int] | tuple[int]) – Output strides of feature from backbone.

• feat_channels (int) – Number of channels for feature.

• out_channels (int) – Number of channels for output.

• num_outs (int) – Number of output scales.

• norm_cfg (ConfigDict or dict) – Config for normalization. Defaults to dict(type=’GN’, num_groups=32).

• act_cfg (ConfigDict or dict) – Config for activation. Defaults to dict(type=’ReLU’).

• encoder (ConfigDict or dict) – Config for transformer encoder. Defaults to None.

• positional_encoding (ConfigDict or dict) – Config for transformer encoder position encoding. Defaults to dict(num_feats=128, normalize=True).

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict. Defaults to None.

forward(feats: List[Tensor]) → Tuple[Tensor, Tensor]

Parameters

feats (list[Tensor]) – Feature maps of each level. Each has shape of (batch_size, c, h, w).

Returns

A tuple containing the following:

• mask_feature (Tensor): shape (batch_size, c, h, w).

• multi_scale_features (list[Tensor]): Multi scale features, each in shape (batch_size, c, h, w).

Return type

tuple

init_weights() → None

Initialize weights.

class mmdet.models.layers.Mask2FormerTransformerDecoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], post_norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, return_intermediate: bool = True, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Decoder of Mask2Former.

class mmdet.models.layers.Mask2FormerTransformerDecoderLayer(self_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, cross_attn_cfg: Optional[Union[ConfigDict, dict]] = {'batch_first': True, 'dropout': 0.0, 'embed_dims': 256, 'num_heads': 8}, ffn_cfg: Optional[Union[ConfigDict, dict]] = {'act_cfg': {'inplace': True, 'type': 'ReLU'}, 'embed_dims': 256, 'feedforward_channels': 1024, 'ffn_drop': 0.0, 'num_fcs': 2}, norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Implements decoder layer in Mask2Former transformer.

forward(query: Tensor, key: Optional[Tensor] = None, value: Optional[Tensor] = None, query_pos: Optional[Tensor] = None, key_pos: Optional[Tensor] = None, self_attn_mask: Optional[Tensor] = None, cross_attn_mask: Optional[Tensor] = None, key_padding_mask: Optional[Tensor] = None, **kwargs) → Tensor

Parameters

• query (Tensor) – The input query, has shape (bs, num_queries, dim).

• key (Tensor, optional) – The input key, has shape (bs, num_keys, dim). If None, the query will be used. Defaults to None.

• value (Tensor, optional) – The input value, has the same shape as key, as in nn.MultiheadAttention.forward. If None, the key will be used. Defaults to None.

• query_pos (Tensor, optional) – The positional encoding for query, has the same shape as query. If not None, it will be added to query before forward function. Defaults to None.

• key_pos (Tensor, optional) – The positional encoding for key, has the same shape as key. If not None, it will be added to key before forward function. If None, and query_pos has the same shape as key, then query_pos will be used for key_pos. Defaults to None.

• self_attn_mask (Tensor, optional) – ByteTensor mask, has shape (num_queries, num_keys), as in nn.MultiheadAttention.forward. Defaults to None.

• cross_attn_mask (Tensor, optional) – ByteTensor mask, has shape (num_queries, num_keys), as in nn.MultiheadAttention.forward. Defaults to None.

• key_padding_mask (Tensor, optional) – The key_padding_mask of self_attn input. ByteTensor, has shape (bs, num_value). Defaults to None.

Returns

forwarded results, has shape (bs, num_queries, dim).

Return type

Tensor

class mmdet.models.layers.Mask2FormerTransformerEncoder(num_layers: int, layer_cfg: Union[ConfigDict, dict], num_cp: int = -1, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Encoder in PixelDecoder of Mask2Former.

forward(query: Tensor, query_pos: Tensor, key_padding_mask: Tensor, spatial_shapes: Tensor, level_start_index: Tensor, valid_ratios: Tensor, reference_points: Tensor, **kwargs) → Tensor

Forward function of Transformer encoder.

Parameters

• query (Tensor) – The input query, has shape (bs, num_queries, dim).

• query_pos (Tensor) – The positional encoding for query, has shape (bs, num_queries, dim). If not None, it will be added to the query before forward function. Defaults to None.

• key_padding_mask (Tensor) – The key_padding_mask of self_attn input. ByteTensor, has shape (bs, num_queries).

• spatial_shapes (Tensor) – Spatial shapes of features in all levels, has shape (num_levels, 2), last dimension represents (h, w).

• level_start_index (Tensor) – The start index of each level. A tensor has shape (num_levels, ) and can be represented as [0, h_0*w_0, h_0*w_0+h_1*w_1, …].

• valid_ratios (Tensor) – The ratios of the valid width and the valid height relative to the width and the height of features in all levels, has shape (bs, num_levels, 2).

• reference_points (Tensor) – The initial reference, has shape (bs, num_queries, 2) with the last dimension arranged as (cx, cy).

Returns

Output queries of Transformer encoder, which is also called ‘encoder output embeddings’ or ‘memory’, has shape (bs, num_queries, dim)

Return type

Tensor

class mmdet.models.layers.NormedConv2d(*args, temperature: float = 20, power: int = 1.0, eps: float = 1e-06, norm_over_kernel: bool = False, **kwargs)

Normalized Conv2d Layer.

Parameters

• tempeature (float, optional) – Tempeature term. Defaults to 20.

• power (int, optional) – Power term. Defaults to 1.0.

• eps (float, optional) – The minimal value of divisor to keep numerical stability. Defaults to 1e-6.

• norm_over_kernel (bool, optional) – Normalize over kernel. Defaults to False.

forward(x: Tensor) → Tensor

Forward function for NormedConv2d.

class mmdet.models.layers.NormedLinear(*args, temperature: float = 20, power: int = 1.0, eps: float = 1e-06, **kwargs)

Normalized Linear Layer.

Parameters

• tempeature (float, optional) – Tempeature term. Defaults to 20.

• power (int, optional) – Power term. Defaults to 1.0.

• eps (float, optional) – The minimal value of divisor to keep numerical stability. Defaults to 1e-6.

forward(x: Tensor) → Tensor

Forward function for NormedLinear.

init_weights() → None

Initialize the weights.

class mmdet.models.layers.PatchEmbed(in_channels: int = 3, embed_dims: int = 768, conv_type: str = 'Conv2d', kernel_size: int = 16, stride: int = 16, padding: Union[int, tuple, str] = 'corner', dilation: int = 1, bias: bool = True, norm_cfg: Optional[Union[ConfigDict, dict]] = None, input_size: Optional[Union[int, tuple]] = None, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Image to Patch Embedding.

We use a conv layer to implement PatchEmbed.

Parameters

• in_channels (int) – The num of input channels. Default: 3

• embed_dims (int) – The dimensions of embedding. Default: 768

• conv_type (str) – The config dict for embedding conv layer type selection. Default: “Conv2d.

• kernel_size (int) – The kernel_size of embedding conv. Default: 16.

• stride (int) – The slide stride of embedding conv. Default: None (Would be set as kernel_size).

• padding (int | tuple | string) – The padding length of embedding conv. When it is a string, it means the mode of adaptive padding, support “same” and “corner” now. Default: “corner”.

• dilation (int) – The dilation rate of embedding conv. Default: 1.

• bias (bool) – Bias of embed conv. Default: True.

• norm_cfg (dict, optional) – Config dict for normalization layer. Default: None.

• input_size (int | tuple | None) – The size of input, which will be used to calculate the out size. Only work when dynamic_size is False. Default: None.

• init_cfg (mmengine.ConfigDict, optional) – The Config for initialization. Default: None.

forward(x: Tensor) → Tuple[Tensor, Tuple[int]]

Parameters

x (Tensor) – Has shape (B, C, H, W). In most case, C is 3.

Returns

Contains merged results and its spatial shape.

• x (Tensor): Has shape (B, out_h * out_w, embed_dims)

• out_size (tuple[int]): Spatial shape of x, arrange as

  (out_h, out_w).

Return type

tuple

class mmdet.models.layers.PatchMerging(in_channels: int, out_channels: int, kernel_size: Optional[Union[int, tuple]] = 2, stride: Optional[Union[int, tuple]] = None, padding: Union[int, tuple, str] = 'corner', dilation: Optional[Union[int, tuple]] = 1, bias: Optional[bool] = False, norm_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'LN'}, init_cfg: Optional[Union[ConfigDict, dict]] = None)

Merge patch feature map.

This layer groups feature map by kernel_size, and applies norm and linear layers to the grouped feature map. Our implementation uses nn.Unfold to merge patch, which is about 25% faster than original implementation. Instead, we need to modify pretrained models for compatibility.

Parameters

• in_channels (int) – The num of input channels. to gets fully covered by filter and stride you specified.. Default: True.

• out_channels (int) – The num of output channels.

• kernel_size (int | tuple, optional) – the kernel size in the unfold layer. Defaults to 2.

• stride (int | tuple, optional) – the stride of the sliding blocks in the unfold layer. Default: None. (Would be set as kernel_size)

• padding (int | tuple | string) – The padding length of embedding conv. When it is a string, it means the mode of adaptive padding, support “same” and “corner” now. Default: “corner”.

• dilation (int | tuple, optional) – dilation parameter in the unfold layer. Default: 1.

• bias (bool, optional) – Whether to add bias in linear layer or not. Defaults: False.

• norm_cfg (dict, optional) – Config dict for normalization layer. Default: dict(type=’LN’).

• init_cfg (dict, optional) – The extra config for initialization. Default: None.

forward(x: Tensor, input_size: Tuple[int]) → Tuple[Tensor, Tuple[int]]

Parameters

• x (Tensor) – Has shape (B, H*W, C_in).

• input_size (tuple[int]) – The spatial shape of x, arrange as (H, W). Default: None.

Returns

Contains merged results and its spatial shape.

• x (Tensor): Has shape (B, Merged_H * Merged_W, C_out)

• out_size (tuple[int]): Spatial shape of x, arrange as

  (Merged_H, Merged_W).

Return type

tuple

class mmdet.models.layers.PixelDecoder(in_channels: Union[List[int], Tuple[int]], feat_channels: int, out_channels: int, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 32, 'type': 'GN'}, act_cfg: Union[ConfigDict, dict] = {'type': 'ReLU'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Pixel decoder with a structure like fpn.

Parameters

• in_channels (list[int] | tuple[int]) – Number of channels in the input feature maps.

• feat_channels (int) – Number channels for feature.

• out_channels (int) – Number channels for output.

• norm_cfg (ConfigDict or dict) – Config for normalization. Defaults to dict(type=’GN’, num_groups=32).

• act_cfg (ConfigDict or dict) – Config for activation. Defaults to dict(type=’ReLU’).

• encoder (ConfigDict or dict) – Config for transorformer encoder.Defaults to None.

• positional_encoding (ConfigDict or dict) – Config for transformer encoder position encoding. Defaults to dict(type=’SinePositionalEncoding’, num_feats=128, normalize=True).

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict. Defaults to None.

forward(feats: List[Tensor], batch_img_metas: List[dict]) → Tuple[Tensor, Tensor]

Parameters

• feats (list[Tensor]) – Feature maps of each level. Each has shape of (batch_size, c, h, w).

• batch_img_metas (list[dict]) – List of image information. Pass in for creating more accurate padding mask. Not used here.

Returns

a tuple containing the following:

• mask_feature (Tensor): Shape (batch_size, c, h, w).

• memory (Tensor): Output of last stage of backbone. Shape (batch_size, c, h, w).

Return type

tuple[Tensor, Tensor]

init_weights() → None

Initialize weights.

class mmdet.models.layers.ResLayer(block: BaseModule, inplanes: int, planes: int, num_blocks: int, stride: int = 1, avg_down: bool = False, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'type': 'BN'}, downsample_first: bool = True, **kwargs)

ResLayer to build ResNet style backbone.

Parameters

• block (nn.Module) – block used to build ResLayer.

• inplanes (int) – inplanes of block.

• planes (int) – planes of block.

• num_blocks (int) – number of blocks.

• stride (int) – stride of the first block. Defaults to 1

• avg_down (bool) – Use AvgPool instead of stride conv when downsampling in the bottleneck. Defaults to False

• conv_cfg (dict) – dictionary to construct and config conv layer. Defaults to None

• norm_cfg (dict) – dictionary to construct and config norm layer. Defaults to dict(type=’BN’)

• downsample_first (bool) – Downsample at the first block or last block. False for Hourglass, True for ResNet. Defaults to True

class mmdet.models.layers.SELayer(channels: int, ratio: int = 16, conv_cfg: Optional[Union[ConfigDict, dict]] = None, act_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = ({'type': 'ReLU'}, {'type': 'Sigmoid'}), init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Squeeze-and-Excitation Module.

Parameters

• channels (int) – The input (and output) channels of the SE layer.

• ratio (int) – Squeeze ratio in SELayer, the intermediate channel will be int(channels/ratio). Defaults to 16.

• conv_cfg (None or dict) – Config dict for convolution layer. Defaults to None, which means using conv2d.

• act_cfg (dict or Sequence[dict]) – Config dict for activation layer. If act_cfg is a dict, two activation layers will be configured by this dict. If act_cfg is a sequence of dicts, the first activation layer will be configured by the first dict and the second activation layer will be configured by the second dict. Defaults to (dict(type=’ReLU’), dict(type=’Sigmoid’))

• init_cfg (dict or list[dict], optional) – Initialization config dict. Defaults to None

forward(x: Tensor) → Tensor

Forward function for SELayer.

class mmdet.models.layers.SiLU(inplace: bool = False)

Applies the Sigmoid Linear Unit (SiLU) function, element-wise.

The SiLU function is also known as the swish function.

\[\text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}\]

Note

See Gaussian Error Linear Units (GELUs) where the SiLU (Sigmoid Linear Unit) was originally coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning and Swish: a Self-Gated Activation Function where the SiLU was experimented with later.

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = nn.SiLU()
>>> input = torch.randn(2)
>>> output = m(input)

extra_repr() → str

Set the extra representation of the module.

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

forward(input: Tensor) → Tensor

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.layers.SimplifiedBasicBlock(inplanes: int, planes: int, stride: int = 1, dilation: int = 1, downsample: Optional[Sequential] = None, style: Union[ConfigDict, dict] = 'pytorch', with_cp: bool = False, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'type': 'BN'}, dcn: Optional[Union[ConfigDict, dict]] = None, plugins: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Simplified version of original basic residual block. This is used in SCNet.

• Norm layer is now optional

• Last ReLU in forward function is removed

forward(x: Tensor) → Tensor

Forward function for SimplifiedBasicBlock.

property norm1: Optional[BaseModule]

normalization layer after the first convolution layer.

Type

nn.Module

property norm2: Optional[BaseModule]

normalization layer after the second convolution layer.

Type

nn.Module

class mmdet.models.layers.SinePositionalEncoding(num_feats: int, temperature: int = 10000, normalize: bool = False, scale: float = 6.283185307179586, eps: float = 1e-06, offset: float = 0.0, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Position encoding with sine and cosine functions.

See End-to-End Object Detection with Transformers for details.

Parameters

• num_feats (int) – The feature dimension for each position along x-axis or y-axis. Note the final returned dimension for each position is 2 times of this value.

• temperature (int, optional) – The temperature used for scaling the position embedding. Defaults to 10000.

• normalize (bool, optional) – Whether to normalize the position embedding. Defaults to False.

• scale (float, optional) – A scale factor that scales the position embedding. The scale will be used only when normalize is True. Defaults to 2*pi.

• eps (float, optional) – A value added to the denominator for numerical stability. Defaults to 1e-6.

• offset (float) – offset add to embed when do the normalization. Defaults to 0.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Defaults to None

forward(mask: Tensor, input: Optional[Tensor] = None) → Tensor

Forward function for SinePositionalEncoding.

Parameters

• mask (Tensor) – ByteTensor mask. Non-zero values representing ignored positions, while zero values means valid positions for this image. Shape [bs, h, w].

• input (Tensor, optional) – Input image/feature Tensor. Shape [bs, c, h, w]

Returns

Returned position embedding with shape

[bs, num_feats*2, h, w].

Return type

pos (Tensor)

class mmdet.models.layers.SinePositionalEncoding3D(num_feats: int, temperature: int = 10000, normalize: bool = False, scale: float = 6.283185307179586, eps: float = 1e-06, offset: float = 0.0, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Position encoding with sine and cosine functions.

See End-to-End Object Detection with Transformers for details.

Parameters

• num_feats (int) – The feature dimension for each position along x-axis or y-axis. Note the final returned dimension for each position is 2 times of this value.

• temperature (int, optional) – The temperature used for scaling the position embedding. Defaults to 10000.

• normalize (bool, optional) – Whether to normalize the position embedding. Defaults to False.

• scale (float, optional) – A scale factor that scales the position embedding. The scale will be used only when normalize is True. Defaults to 2*pi.

• eps (float, optional) – A value added to the denominator for numerical stability. Defaults to 1e-6.

• offset (float) – offset add to embed when do the normalization. Defaults to 0.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Defaults to None.

forward(mask: Tensor) → Tensor

Forward function for SinePositionalEncoding3D.

Parameters

mask (Tensor) – ByteTensor mask. Non-zero values representing ignored positions, while zero values means valid positions for this image. Shape [bs, t, h, w].

Returns

Returned position embedding with shape

[bs, num_feats*2, h, w].

Return type

pos (Tensor)

class mmdet.models.layers.TransformerEncoderPixelDecoder(in_channels: Union[List[int], Tuple[int]], feat_channels: int, out_channels: int, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 32, 'type': 'GN'}, act_cfg: Union[ConfigDict, dict] = {'type': 'ReLU'}, encoder: Optional[Union[ConfigDict, dict]] = None, positional_encoding: Union[ConfigDict, dict] = {'normalize': True, 'num_feats': 128}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Pixel decoder with transformer encoder inside.

Parameters

• in_channels (list[int] | tuple[int]) – Number of channels in the input feature maps.

• feat_channels (int) – Number channels for feature.

• out_channels (int) – Number channels for output.

• norm_cfg (ConfigDict or dict) – Config for normalization. Defaults to dict(type=’GN’, num_groups=32).

• act_cfg (ConfigDict or dict) – Config for activation. Defaults to dict(type=’ReLU’).

• encoder (ConfigDict or dict) – Config for transformer encoder. Defaults to None.

• positional_encoding (ConfigDict or dict) – Config for transformer encoder position encoding. Defaults to dict(num_feats=128, normalize=True).

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict. Defaults to None.

forward(feats: List[Tensor], batch_img_metas: List[dict]) → Tuple[Tensor, Tensor]

Parameters

• feats (list[Tensor]) – Feature maps of each level. Each has shape of (batch_size, c, h, w).

• batch_img_metas (list[dict]) – List of image information. Pass in for creating more accurate padding mask.

Returns

a tuple containing the following:

• mask_feature (Tensor): shape (batch_size, c, h, w).

• memory (Tensor): shape (batch_size, c, h, w).

Return type

tuple

init_weights() → None

Initialize weights.

mmdet.models.layers.adaptive_avg_pool2d(input, output_size)

Handle empty batch dimension to adaptive_avg_pool2d.

Parameters

• input (tensor) – 4D tensor.

• output_size (int, tuple[int,int]) – the target output size.

mmdet.models.layers.coordinate_to_encoding(coord_tensor: Tensor, num_feats: int = 128, temperature: int = 10000, scale: float = 6.283185307179586)

Convert coordinate tensor to positional encoding.

Parameters

• coord_tensor (Tensor) – Coordinate tensor to be converted to positional encoding. With the last dimension as 2 or 4.

• num_feats (int, optional) – The feature dimension for each position along x-axis or y-axis. Note the final returned dimension for each position is 2 times of this value. Defaults to 128.

• temperature (int, optional) – The temperature used for scaling the position embedding. Defaults to 10000.

• scale (float, optional) – A scale factor that scales the position embedding. The scale will be used only when normalize is True. Defaults to 2*pi.

Returns

Returned encoded positional tensor.

Return type

Tensor

mmdet.models.layers.fast_nms(multi_bboxes: Tensor, multi_scores: Tensor, multi_coeffs: Tensor, score_thr: float, iou_thr: float, top_k: int, max_num: int = -1) → Union[Tuple[Tensor, Tensor, Tensor], Tuple[Tensor, Tensor]]

Fast NMS in YOLACT.

Fast NMS allows already-removed detections to suppress other detections so that every instance can be decided to be kept or discarded in parallel, which is not possible in traditional NMS. This relaxation allows us to implement Fast NMS entirely in standard GPU-accelerated matrix operations.

Parameters

• multi_bboxes (Tensor) – shape (n, #class*4) or (n, 4)

• multi_scores (Tensor) – shape (n, #class+1), where the last column contains scores of the background class, but this will be ignored.

• multi_coeffs (Tensor) – shape (n, #class*coeffs_dim).

• score_thr (float) – bbox threshold, bboxes with scores lower than it will not be considered.

• iou_thr (float) – IoU threshold to be considered as conflicted.

• top_k (int) – if there are more than top_k bboxes before NMS, only top top_k will be kept.

• max_num (int) – if there are more than max_num bboxes after NMS, only top max_num will be kept. If -1, keep all the bboxes. Default: -1.

Returns

(dets, labels, coefficients), tensors of shape (k, 5), (k, 1), and (k, coeffs_dim). Dets are boxes with scores. Labels are 0-based.

Return type

Union[Tuple[Tensor, Tensor, Tensor], Tuple[Tensor, Tensor]]

mmdet.models.layers.inverse_sigmoid(x: Tensor, eps: float = 1e-05) → Tensor

Inverse function of sigmoid.

Parameters

• x (Tensor) – The tensor to do the inverse.

• eps (float) – EPS avoid numerical overflow. Defaults 1e-5.

Returns

The x has passed the inverse function of sigmoid, has the same shape with input.

Return type

Tensor

mmdet.models.layers.mask_matrix_nms(masks, labels, scores, filter_thr=-1, nms_pre=-1, max_num=-1, kernel='gaussian', sigma=2.0, mask_area=None)

Matrix NMS for multi-class masks.

Parameters

• masks (Tensor) – Has shape (num_instances, h, w)

• labels (Tensor) – Labels of corresponding masks, has shape (num_instances,).

• scores (Tensor) – Mask scores of corresponding masks, has shape (num_instances).

• filter_thr (float) – Score threshold to filter the masks after matrix nms. Default: -1, which means do not use filter_thr.

• nms_pre (int) – The max number of instances to do the matrix nms. Default: -1, which means do not use nms_pre.

• max_num (int, optional) – If there are more than max_num masks after matrix, only top max_num will be kept. Default: -1, which means do not use max_num.

• kernel (str) – ‘linear’ or ‘gaussian’.

• sigma (float) – std in gaussian method.

• mask_area (Tensor) – The sum of seg_masks.

Returns

Processed mask results.

• scores (Tensor): Updated scores, has shape (n,).

• labels (Tensor): Remained labels, has shape (n,).

• masks (Tensor): Remained masks, has shape (n, w, h).

• keep_inds (Tensor): The indices number of

  the remaining mask in the input mask, has shape (n,).

Return type

tuple(Tensor)

mmdet.models.layers.multiclass_nms(multi_bboxes: Tensor, multi_scores: Tensor, score_thr: float, nms_cfg: Union[ConfigDict, dict], max_num: int = -1, score_factors: Optional[Tensor] = None, return_inds: bool = False, box_dim: int = 4) → Union[Tuple[Tensor, Tensor, Tensor], Tuple[Tensor, Tensor]]

NMS for multi-class bboxes.

Parameters

• multi_bboxes (Tensor) – shape (n, #class*4) or (n, 4)

• multi_scores (Tensor) – shape (n, #class), where the last column contains scores of the background class, but this will be ignored.

• score_thr (float) – bbox threshold, bboxes with scores lower than it will not be considered.

• nms_cfg (Union[ConfigDict, dict]) – a dict that contains the arguments of nms operations.

• max_num (int, optional) – if there are more than max_num bboxes after NMS, only top max_num will be kept. Default to -1.

• score_factors (Tensor, optional) – The factors multiplied to scores before applying NMS. Default to None.

• return_inds (bool, optional) – Whether return the indices of kept bboxes. Default to False.

• box_dim (int) – The dimension of boxes. Defaults to 4.

Returns

(dets, labels, indices (optional)), tensors of shape (k, 5), (k), and (k). Dets are boxes with scores. Labels are 0-based.

Return type

Union[Tuple[Tensor, Tensor, Tensor], Tuple[Tensor, Tensor]]

mmdet.models.layers.nchw_to_nlc(x)

Flatten [N, C, H, W] shape tensor to [N, L, C] shape tensor.

Parameters

x (Tensor) – The input tensor of shape [N, C, H, W] before conversion.

Returns

The output tensor of shape [N, L, C] after conversion.

Return type

Tensor

mmdet.models.layers.nlc_to_nchw(x: Tensor, hw_shape: Sequence[int]) → Tensor

Convert [N, L, C] shape tensor to [N, C, H, W] shape tensor.

Parameters

• x (Tensor) – The input tensor of shape [N, L, C] before conversion.

• hw_shape (Sequence[int]) – The height and width of output feature map.

Returns

The output tensor of shape [N, C, H, W] after conversion.

Return type

Tensor

losses

class mmdet.models.losses.Accuracy(topk=(1,), thresh=None)

forward(pred, target)

Forward function to calculate accuracy.

Parameters

• pred (torch.Tensor) – Prediction of models.

• target (torch.Tensor) – Target for each prediction.

Returns

The accuracies under different topk criterions.

Return type

tuple[float]

class mmdet.models.losses.AssociativeEmbeddingLoss(pull_weight=0.25, push_weight=0.25)

Associative Embedding Loss.

More details can be found in Associative Embedding and CornerNet . Code is modified from kp_utils.py # noqa: E501

Parameters

• pull_weight (float) – Loss weight for corners from same object.

• push_weight (float) – Loss weight for corners from different object.

forward(pred, target, match)

Forward function.

class mmdet.models.losses.BalancedL1Loss(alpha=0.5, gamma=1.5, beta=1.0, reduction='mean', loss_weight=1.0)

Balanced L1 Loss.

arXiv: https://arxiv.org/pdf/1904.02701.pdf (CVPR 2019)

Parameters

• alpha (float) – The denominator alpha in the balanced L1 loss. Defaults to 0.5.

• gamma (float) – The gamma in the balanced L1 loss. Defaults to 1.5.

• beta (float, optional) – The loss is a piecewise function of prediction and target. beta serves as a threshold for the difference between the prediction and target. Defaults to 1.0.

• reduction (str, optional) – The method that reduces the loss to a scalar. Options are “none”, “mean” and “sum”.

• loss_weight (float, optional) – The weight of the loss. Defaults to 1.0

forward(pred, target, weight=None, avg_factor=None, reduction_override=None, **kwargs)

Forward function of loss.

Parameters

• pred (torch.Tensor) – The prediction with shape (N, 4).

• target (torch.Tensor) – The learning target of the prediction with shape (N, 4).

• weight (torch.Tensor, optional) – Sample-wise loss weight with shape (N, ).

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”.

Returns

The calculated loss

Return type

torch.Tensor

class mmdet.models.losses.BoundedIoULoss(beta: float = 0.2, eps: float = 0.001, reduction: str = 'mean', loss_weight: float = 1.0)

BIoULoss.

This is an implementation of paper Improving Object Localization with Fitness NMS and Bounded IoU Loss..

Parameters

• beta (float, optional) – Beta parameter in smoothl1.

• eps (float, optional) – Epsilon to avoid NaN values.

• reduction (str) – Options are “none”, “mean” and “sum”.

• loss_weight (float) – Weight of loss.

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None, **kwargs) → Tensor

Forward function.

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – The learning target of the prediction, shape (n, 4).

• weight (Optional[Tensor], optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (Optional[int], optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (Optional[str], optional) – The reduction method used to override the original reduction method of the loss. Defaults to None. Options are “none”, “mean” and “sum”.

Returns

Loss tensor.

Return type

Tensor

class mmdet.models.losses.CIoULoss(eps: float = 1e-06, reduction: str = 'mean', loss_weight: float = 1.0)

Implementation of paper `Enhancing Geometric Factors into Model Learning and Inference for Object Detection and Instance Segmentation.

Code is modified from https://github.com/Zzh-tju/CIoU.

Parameters

• eps (float) – Epsilon to avoid log(0).

• reduction (str) – Options are “none”, “mean” and “sum”.

• loss_weight (float) – Weight of loss.

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None, **kwargs) → Tensor

Forward function.

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – The learning target of the prediction, shape (n, 4).

• weight (Optional[Tensor], optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (Optional[int], optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (Optional[str], optional) – The reduction method used to override the original reduction method of the loss. Defaults to None. Options are “none”, “mean” and “sum”.

Returns

Loss tensor.

Return type

Tensor

class mmdet.models.losses.CrossEntropyCustomLoss(use_sigmoid=False, use_mask=False, reduction='mean', num_classes=-1, class_weight=None, ignore_index=None, loss_weight=1.0, avg_non_ignore=False)

class mmdet.models.losses.CrossEntropyLoss(use_sigmoid=False, use_mask=False, reduction='mean', class_weight=None, ignore_index=None, loss_weight=1.0, avg_non_ignore=False)

extra_repr()

Extra repr.

forward(cls_score, label, weight=None, avg_factor=None, reduction_override=None, ignore_index=None, **kwargs)

Forward function.

Parameters

• cls_score (torch.Tensor) – The prediction.

• label (torch.Tensor) – The learning label of the prediction.

• weight (torch.Tensor, optional) – Sample-wise loss weight.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The method used to reduce the loss. Options are “none”, “mean” and “sum”.

• ignore_index (int | None) – The label index to be ignored. If not None, it will override the default value. Default: None.

Returns

The calculated loss.

Return type

torch.Tensor

class mmdet.models.losses.DDQAuxLoss(loss_cls={'activated': True, 'beta': 2.0, 'loss_weight': 1.0, 'type': 'QualityFocalLoss', 'use_sigmoid': True}, loss_bbox={'loss_weight': 2.0, 'type': 'GIoULoss'}, train_cfg={'alpha': 1, 'assigner': {'topk': 8, 'type': 'TopkHungarianAssigner'}, 'beta': 6})

DDQ auxiliary branches loss for dense queries.

Parameters

• loss_cls (dict) – Configuration of classification loss function.

• loss_bbox (dict) – Configuration of bbox regression loss function.

• train_cfg (dict) – Configuration of gt targets assigner for each predicted bbox.

get_targets(cls_scores, bbox_preds, gt_bboxes_list, img_metas, gt_labels_list=None, **kwargs)

Compute regression and classification targets for a batch images.

Parameters

• cls_scores (Tensor) – Predicted normalized classification scores, has shape (bs, num_dense_queries, cls_out_channels).

• bbox_preds (Tensor) – Predicted unnormalized bbox coordinates, has shape (bs, num_dense_queries, 4) with the last dimension arranged as (x1, y1, x2, y2).

• gt_bboxes_list (List[Tensor]) – List of unnormalized ground truth bboxes for each image, each has shape (num_gt, 4) with the last dimension arranged as (x1, y1, x2, y2). NOTE: num_gt is dynamic for each image.

• img_metas (list[dict]) – Meta information for one image, e.g., image size, scaling factor, etc.

• gt_labels_list (list[Tensor]) – List of ground truth classification index for each image, each has shape (num_gt,). NOTE: num_gt is dynamic for each image. Default: None.

Returns

a tuple containing the following targets.

• all_labels (list[Tensor]): Labels for all images.

• all_label_weights (list[Tensor]): Label weights for all images.

• all_bbox_targets (list[Tensor]): Bbox targets for all images.

• all_assign_metrics (list[Tensor]): Normalized alignment metrics

  for all images.

Return type

tuple

loss(cls_scores, bbox_preds, gt_bboxes, gt_labels, img_metas, **kwargs)

Calculate auxiliary branches loss for dense queries.

Parameters

• cls_scores (Tensor) – Predicted normalized classification scores, has shape (bs, num_dense_queries, cls_out_channels).

• bbox_preds (Tensor) – Predicted unnormalized bbox coordinates, has shape (bs, num_dense_queries, 4) with the last dimension arranged as (x1, y1, x2, y2).

• gt_bboxes (list[Tensor]) – List of unnormalized ground truth bboxes for each image, each has shape (num_gt, 4) with the last dimension arranged as (x1, y1, x2, y2). NOTE: num_gt is dynamic for each image.

• gt_labels (list[Tensor]) – List of ground truth classification index for each image, each has shape (num_gt,). NOTE: num_gt is dynamic for each image.

• img_metas (list[dict]) – Meta information for one image, e.g., image size, scaling factor, etc.

Returns

A dictionary of loss components.

Return type

dict

loss_single(cls_score, bbox_pred, labels, label_weights, bbox_targets, alignment_metrics)

Calculate auxiliary branches loss for dense queries for one image.

Parameters

• cls_score (Tensor) – Predicted normalized classification scores for one image, has shape (num_dense_queries, cls_out_channels).

• bbox_pred (Tensor) – Predicted unnormalized bbox coordinates for one image, has shape (num_dense_queries, 4) with the last dimension arranged as (x1, y1, x2, y2).

• labels (Tensor) – Labels for one image.

• label_weights (Tensor) – Label weights for one image.

• bbox_targets (Tensor) – Bbox targets for one image.

• alignment_metrics (Tensor) – Normalized alignment metrics for one image.

Returns

A tuple of loss components and loss weights.

Return type

tuple

class mmdet.models.losses.DIoULoss(eps: float = 1e-06, reduction: str = 'mean', loss_weight: float = 1.0)

Implementation of `Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression https://arxiv.org/abs/1911.08287`_.

Code is modified from https://github.com/Zzh-tju/DIoU.

Parameters

• eps (float) – Epsilon to avoid log(0).

• reduction (str) – Options are “none”, “mean” and “sum”.

• loss_weight (float) – Weight of loss.

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None, **kwargs) → Tensor

Forward function.

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – The learning target of the prediction, shape (n, 4).

• weight (Optional[Tensor], optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (Optional[int], optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (Optional[str], optional) – The reduction method used to override the original reduction method of the loss. Defaults to None. Options are “none”, “mean” and “sum”.

Returns

Loss tensor.

Return type

Tensor

class mmdet.models.losses.DiceLoss(use_sigmoid=True, activate=True, reduction='mean', naive_dice=False, loss_weight=1.0, eps=0.001)

forward(pred, target, weight=None, reduction_override=None, avg_factor=None)

Forward function.

Parameters

• pred (torch.Tensor) – The prediction, has a shape (n, *).

• target (torch.Tensor) – The label of the prediction, shape (n, *), same shape of pred.

• weight (torch.Tensor, optional) – The weight of loss for each prediction, has a shape (n,). Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”.

Returns

The calculated loss

Return type

torch.Tensor

class mmdet.models.losses.DistributionFocalLoss(reduction='mean', loss_weight=1.0)

Distribution Focal Loss (DFL) is a variant of Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection.

Parameters

• reduction (str) – Options are ‘none’, ‘mean’ and ‘sum’.

• loss_weight (float) – Loss weight of current loss.

forward(pred, target, weight=None, avg_factor=None, reduction_override=None)

Forward function.

Parameters

• pred (torch.Tensor) – Predicted general distribution of bounding boxes (before softmax) with shape (N, n+1), n is the max value of the integral set {0, …, n} in paper.

• target (torch.Tensor) – Target distance label for bounding boxes with shape (N,).

• weight (torch.Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

class mmdet.models.losses.EIoULoss(eps: float = 1e-06, reduction: str = 'mean', loss_weight: float = 1.0, smooth_point: float = 0.1)

Implementation of paper Extended-IoU Loss: A Systematic IoU-Related Method: Beyond Simplified Regression for Better Localization

Code is modified from https://github.com//ShiqiYu/libfacedetection.train.

Parameters

• eps (float) – Epsilon to avoid log(0).

• reduction (str) – Options are “none”, “mean” and “sum”.

• loss_weight (float) – Weight of loss.

• smooth_point (float) – hyperparameter, default is 0.1.

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None, **kwargs) → Tensor

Forward function.

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – The learning target of the prediction, shape (n, 4).

• weight (Optional[Tensor], optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (Optional[int], optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (Optional[str], optional) – The reduction method used to override the original reduction method of the loss. Defaults to None. Options are “none”, “mean” and “sum”.

Returns

Loss tensor.

Return type

Tensor

class mmdet.models.losses.EQLV2Loss(use_sigmoid: bool = True, reduction: str = 'mean', class_weight: Optional[Tensor] = None, loss_weight: float = 1.0, num_classes: int = 1203, use_distributed: bool = False, mu: float = 0.8, alpha: float = 4.0, gamma: int = 12, vis_grad: bool = False, test_with_obj: bool = True)

forward(cls_score: Tensor, label: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[Tensor] = None) → Tensor

Equalization Loss v2

Parameters

• cls_score (Tensor) – The prediction with shape (N, C), C is the number of classes.

• label (Tensor) – The ground truth label of the predicted target with shape (N, C), C is the number of classes.

• weight (Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”.

Returns

The calculated loss

Return type

Tensor

class mmdet.models.losses.FocalCustomLoss(use_sigmoid=True, num_classes=-1, gamma=2.0, alpha=0.25, reduction='mean', loss_weight=1.0, activated=False)

forward(pred, target, weight=None, avg_factor=None, reduction_override=None)

Forward function.

Parameters

• pred (torch.Tensor) – The prediction.

• target (torch.Tensor) – The learning label of the prediction.

• weight (torch.Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”.

Returns

The calculated loss

Return type

torch.Tensor

class mmdet.models.losses.FocalLoss(use_sigmoid=True, gamma=2.0, alpha=0.25, reduction='mean', loss_weight=1.0, activated=False)

forward(pred, target, weight=None, avg_factor=None, reduction_override=None)

Forward function.

Parameters

• pred (torch.Tensor) – The prediction.

• target (torch.Tensor) – The learning label of the prediction. The target shape support (N,C) or (N,), (N,C) means one-hot form.

• weight (torch.Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”.

Returns

The calculated loss

Return type

torch.Tensor

class mmdet.models.losses.GHMC(bins=10, momentum=0, use_sigmoid=True, loss_weight=1.0, reduction='mean')

GHM Classification Loss.

Details of the theorem can be viewed in the paper Gradient Harmonized Single-stage Detector.

Parameters

• bins (int) – Number of the unit regions for distribution calculation.

• momentum (float) – The parameter for moving average.

• use_sigmoid (bool) – Can only be true for BCE based loss now.

• loss_weight (float) – The weight of the total GHM-C loss.

• reduction (str) – Options are “none”, “mean” and “sum”. Defaults to “mean”

forward(pred, target, label_weight, reduction_override=None, **kwargs)

Calculate the GHM-C loss.

Parameters

• pred (float tensor of size [batch_num, class_num]) – The direct prediction of classification fc layer.

• target (float tensor of size [batch_num, class_num]) – Binary class target for each sample.

• label_weight (float tensor of size [batch_num, class_num]) – the value is 1 if the sample is valid and 0 if ignored.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

Returns

The gradient harmonized loss.

class mmdet.models.losses.GHMR(mu=0.02, bins=10, momentum=0, loss_weight=1.0, reduction='mean')

GHM Regression Loss.

Details of the theorem can be viewed in the paper Gradient Harmonized Single-stage Detector.

Parameters

• mu (float) – The parameter for the Authentic Smooth L1 loss.

• bins (int) – Number of the unit regions for distribution calculation.

• momentum (float) – The parameter for moving average.

• loss_weight (float) – The weight of the total GHM-R loss.

• reduction (str) – Options are “none”, “mean” and “sum”. Defaults to “mean”

forward(pred, target, label_weight, avg_factor=None, reduction_override=None)

Calculate the GHM-R loss.

Parameters

• pred (float tensor of size [batch_num, 4 (* class_num)]) – The prediction of box regression layer. Channel number can be 4 or 4 * class_num depending on whether it is class-agnostic.

• target (float tensor of size [batch_num, 4 (* class_num)]) – The target regression values with the same size of pred.

• label_weight (float tensor of size [batch_num, 4 (* class_num)]) – The weight of each sample, 0 if ignored.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

Returns

The gradient harmonized loss.

class mmdet.models.losses.GIoULoss(eps: float = 1e-06, reduction: str = 'mean', loss_weight: float = 1.0)

Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression.

Parameters

• eps (float) – Epsilon to avoid log(0).

• reduction (str) – Options are “none”, “mean” and “sum”.

• loss_weight (float) – Weight of loss.

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None, **kwargs) → Tensor

Forward function.

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – The learning target of the prediction, shape (n, 4).

• weight (Optional[Tensor], optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (Optional[int], optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (Optional[str], optional) – The reduction method used to override the original reduction method of the loss. Defaults to None. Options are “none”, “mean” and “sum”.

Returns

Loss tensor.

Return type

Tensor

class mmdet.models.losses.GaussianFocalLoss(alpha: float = 2.0, gamma: float = 4.0, reduction: str = 'mean', loss_weight: float = 1.0, pos_weight: float = 1.0, neg_weight: float = 1.0)

GaussianFocalLoss is a variant of focal loss.

More details can be found in the paper Code is modified from kp_utils.py # noqa: E501 Please notice that the target in GaussianFocalLoss is a gaussian heatmap, not 0/1 binary target.

Parameters

• alpha (float) – Power of prediction.

• gamma (float) – Power of target for negative samples.

• reduction (str) – Options are “none”, “mean” and “sum”.

• loss_weight (float) – Loss weight of current loss.

• pos_weight (float) – Positive sample loss weight. Defaults to 1.0.

• neg_weight (float) – Negative sample loss weight. Defaults to 1.0.

forward(pred: Tensor, target: Tensor, pos_inds: Optional[Tensor] = None, pos_labels: Optional[Tensor] = None, weight: Optional[Tensor] = None, avg_factor: Optional[Union[int, float]] = None, reduction_override: Optional[str] = None) → Tensor

Forward function.

If you want to manually determine which positions are positive samples, you can set the pos_index and pos_label parameter. Currently, only the CenterNet update version uses the parameter.

Parameters

• pred (torch.Tensor) – The prediction. The shape is (N, num_classes).

• target (torch.Tensor) – The learning target of the prediction in gaussian distribution. The shape is (N, num_classes).

• pos_inds (torch.Tensor) – The positive sample index. Defaults to None.

• pos_labels (torch.Tensor) – The label corresponding to the positive sample index. Defaults to None.

• weight (torch.Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, float, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

class mmdet.models.losses.IoULoss(linear: bool = False, eps: float = 1e-06, reduction: str = 'mean', loss_weight: float = 1.0, mode: str = 'log')

IoULoss.

Computing the IoU loss between a set of predicted bboxes and target bboxes.

Parameters

• linear (bool) – If True, use linear scale of loss else determined by mode. Default: False.

• eps (float) – Epsilon to avoid log(0).

• reduction (str) – Options are “none”, “mean” and “sum”.

• loss_weight (float) – Weight of loss.

• mode (str) – Loss scaling mode, including “linear”, “square”, and “log”. Default: ‘log’

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None, **kwargs) → Tensor

Forward function.

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – The learning target of the prediction, shape (n, 4).

• weight (Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None. Options are “none”, “mean” and “sum”.

Returns

Loss tensor.

Return type

Tensor

class mmdet.models.losses.KnowledgeDistillationKLDivLoss(reduction: str = 'mean', loss_weight: float = 1.0, T: int = 10)

Loss function for knowledge distilling using KL divergence.

Parameters

• reduction (str) – Options are ‘none’, ‘mean’ and ‘sum’.

• loss_weight (float) – Loss weight of current loss.

• T (int) – Temperature for distillation.

forward(pred: Tensor, soft_label: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None) → Tensor

Forward function.

Parameters

• pred (Tensor) – Predicted logits with shape (N, n + 1).

• soft_label (Tensor) – Target logits with shape (N, N + 1).

• weight (Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

Returns

Loss tensor.

Return type

Tensor

class mmdet.models.losses.L1Loss(reduction: str = 'mean', loss_weight: float = 1.0)

L1 loss.

Parameters

• reduction (str, optional) – The method to reduce the loss. Options are “none”, “mean” and “sum”.

• loss_weight (float, optional) – The weight of loss.

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None) → Tensor

Forward function.

Parameters

• pred (Tensor) – The prediction.

• target (Tensor) – The learning target of the prediction.

• weight (Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

Returns

Calculated loss

Return type

Tensor

class mmdet.models.losses.L2Loss(neg_pos_ub: int = -1, pos_margin: float = -1, neg_margin: float = -1, hard_mining: bool = False, reduction: str = 'mean', loss_weight: float = 1.0)

L2 loss.

Parameters

• reduction (str, optional) – The method to reduce the loss. Options are “none”, “mean” and “sum”.

• loss_weight (float, optional) – The weight of loss.

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[float] = None, reduction_override: Optional[str] = None) → Tensor

Forward function.

Parameters

• pred (torch.Tensor) – The prediction.

• target (torch.Tensor) – The learning target of the prediction.

• weight (torch.Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (float, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

static random_choice(gallery: Union[list, ndarray, Tensor], num: int) → ndarray

Random select some elements from the gallery.

It seems that Pytorch’s implementation is slower than numpy so we use numpy to randperm the indices.

update_weight(pred: Tensor, target: Tensor, weight: Tensor, avg_factor: float) → Tuple[Tensor, Tensor, float]

Update the weight according to targets.

class mmdet.models.losses.MSELoss(reduction: str = 'mean', loss_weight: float = 1.0)

MSELoss.

Parameters

• reduction (str, optional) – The method that reduces the loss to a scalar. Options are “none”, “mean” and “sum”.

• loss_weight (float, optional) – The weight of the loss. Defaults to 1.0

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None) → Tensor

Forward function of loss.

Parameters

• pred (Tensor) – The prediction.

• target (Tensor) – The learning target of the prediction.

• weight (Tensor, optional) – Weight of the loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

Returns

The calculated loss.

Return type

Tensor

class mmdet.models.losses.MarginL2Loss(neg_pos_ub: int = -1, pos_margin: float = -1, neg_margin: float = -1, hard_mining: bool = False, reduction: str = 'mean', loss_weight: float = 1.0)

L2 loss with margin.

Parameters

• neg_pos_ub (int, optional) – The upper bound of negative to positive samples in hard mining. Defaults to -1.

• pos_margin (float, optional) – The similarity margin for positive samples in hard mining. Defaults to -1.

• neg_margin (float, optional) – The similarity margin for negative samples in hard mining. Defaults to -1.

• hard_mining (bool, optional) – Whether to use hard mining. Defaults to False.

• reduction (str, optional) – The method to reduce the loss. Options are “none”, “mean” and “sum”. Defaults to “mean”.

• loss_weight (float, optional) – The weight of loss. Defaults to 1.0.

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[float] = None, reduction_override: Optional[str] = None) → Tensor

Forward function.

Parameters

• pred (torch.Tensor) – The prediction.

• target (torch.Tensor) – The learning target of the prediction.

• weight (torch.Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (float, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

static random_choice(gallery: Union[list, ndarray, Tensor], num: int) → ndarray

Random select some elements from the gallery.

It seems that Pytorch’s implementation is slower than numpy so we use numpy to randperm the indices.

Parameters

• gallery (list | np.ndarray | torch.Tensor) – The gallery from which to sample.

• num (int) – The number of elements to sample.

update_weight(pred: Tensor, target: Tensor, weight: Tensor, avg_factor: float) → Tuple[Tensor, Tensor, float]

Update the weight according to targets.

Parameters

• pred (torch.Tensor) – The prediction.

• target (torch.Tensor) – The learning target of the prediction.

• weight (torch.Tensor) – The weight of loss for each prediction.

• avg_factor (float) – Average factor that is used to average the loss.

Returns

The updated prediction, weight and average factor.

Return type

tuple[torch.Tensor]

class mmdet.models.losses.MultiPosCrossEntropyLoss(reduction: str = 'mean', loss_weight: float = 1.0)

Multi-positive targets cross entropy loss.

Parameters

• reduction (str, optional) – The method to reduce the loss. Options are “none”, “mean” and “sum”. Defaults to “mean”.

• loss_weight (float, optional) – The weight of loss. Defaults to 1.0.

forward(cls_score: Tensor, label: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[float] = None, reduction_override: Optional[str] = None, **kwargs) → Tensor

Forward function.

Parameters

• cls_score (torch.Tensor) – The classification score.

• label (torch.Tensor) – The assigned label of the prediction.

• weight (torch.Tensor) – The element-wise weight.

• avg_factor (float) – Average factor when computing the mean of losses.

• reduction_override (str) – Same as built-in losses of PyTorch.

Returns

Calculated loss

Return type

torch.Tensor

multi_pos_cross_entropy(pred: Tensor, label: Tensor, weight: Optional[Tensor] = None, reduction: str = 'mean', avg_factor: Optional[float] = None) → Tensor

Multi-positive targets cross entropy loss.

Parameters

• pred (torch.Tensor) – The prediction.

• label (torch.Tensor) – The assigned label of the prediction.

• weight (torch.Tensor) – The element-wise weight.

• reduction (str) – Same as built-in losses of PyTorch.

• avg_factor (float) – Average factor when computing the mean of losses.

Returns

Calculated loss

Return type

torch.Tensor

class mmdet.models.losses.QualityFocalLoss(use_sigmoid=True, beta=2.0, reduction='mean', loss_weight=1.0, activated=False)

Quality Focal Loss (QFL) is a variant of Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection.

Parameters

• use_sigmoid (bool) – Whether sigmoid operation is conducted in QFL. Defaults to True.

• beta (float) – The beta parameter for calculating the modulating factor. Defaults to 2.0.

• reduction (str) – Options are “none”, “mean” and “sum”.

• loss_weight (float) – Loss weight of current loss.

• activated (bool, optional) – Whether the input is activated. If True, it means the input has been activated and can be treated as probabilities. Else, it should be treated as logits. Defaults to False.

forward(pred, target, weight=None, avg_factor=None, reduction_override=None)

Forward function.

Parameters

• pred (torch.Tensor) – Predicted joint representation of classification and quality (IoU) estimation with shape (N, C), C is the number of classes.

• target (Union(tuple([torch.Tensor]),Torch.Tensor)) – The type is tuple, it should be included Target category label with shape (N,) and target quality label with shape (N,).The type is torch.Tensor, the target should be one-hot form with soft weights.

• weight (torch.Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

class mmdet.models.losses.SIoULoss(eps: float = 1e-06, reduction: str = 'mean', loss_weight: float = 1.0, neg_gamma: bool = False)

Implementation of paper `SIoU Loss: More Powerful Learning for Bounding Box Regression.

Code is modified from https://github.com/meituan/YOLOv6.

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – Corresponding gt bboxes, shape (n, 4).

• eps (float) – Eps to avoid log(0).

• neg_gamma (bool) – True follows original implementation in paper.

Returns

Loss tensor.

Return type

Tensor

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None, **kwargs) → Tensor

Forward function.

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – The learning target of the prediction, shape (n, 4).

• weight (Optional[Tensor], optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (Optional[int], optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (Optional[str], optional) – The reduction method used to override the original reduction method of the loss. Defaults to None. Options are “none”, “mean” and “sum”.

Returns

Loss tensor.

Return type

Tensor

class mmdet.models.losses.SeesawLoss(use_sigmoid: bool = False, p: float = 0.8, q: float = 2.0, num_classes: int = 1203, eps: float = 0.01, reduction: str = 'mean', loss_weight: float = 1.0, return_dict: bool = True)

Seesaw Loss for Long-Tailed Instance Segmentation (CVPR 2021) arXiv: https://arxiv.org/abs/2008.10032

Parameters

• use_sigmoid (bool, optional) – Whether the prediction uses sigmoid of softmax. Only False is supported.

• p (float, optional) – The p in the mitigation factor. Defaults to 0.8.

• q (float, optional) – The q in the compenstation factor. Defaults to 2.0.

• num_classes (int, optional) – The number of classes. Default to 1203 for LVIS v1 dataset.

• eps (float, optional) – The minimal value of divisor to smooth the computation of compensation factor

• reduction (str, optional) – The method that reduces the loss to a scalar. Options are “none”, “mean” and “sum”.

• loss_weight (float, optional) – The weight of the loss. Defaults to 1.0

• return_dict (bool, optional) – Whether return the losses as a dict. Default to True.

forward(cls_score: Tensor, labels: Tensor, label_weights: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None) → Union[Tensor, Dict[str, Tensor]]

Forward function.

Parameters

• cls_score (Tensor) – The prediction with shape (N, C + 2).

• labels (Tensor) – The learning label of the prediction.

• label_weights (Tensor, optional) – Sample-wise loss weight.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction (str, optional) – The method used to reduce the loss. Options are “none”, “mean” and “sum”.

Returns

if return_dict == False: The calculated loss | if return_dict == True: The dict of calculated losses for objectness and classes, respectively.

Return type

Tensor | Dict [str, Tensor]

get_accuracy(cls_score: Tensor, labels: Tensor) → Dict[str, Tensor]

Get custom accuracy w.r.t. cls_score and labels.

Parameters

• cls_score (Tensor) – The prediction with shape (N, C + 2).

• labels (Tensor) – The learning label of the prediction.

Returns

The accuracy for objectness and classes,

respectively.

Return type

Dict [str, Tensor]

get_activation(cls_score: Tensor) → Tensor

Get custom activation of cls_score.

Parameters

cls_score (Tensor) – The prediction with shape (N, C + 2).

Returns

The custom activation of cls_score with shape

(N, C + 1).

Return type

Tensor

get_cls_channels(num_classes: int) → int

Get custom classification channels.

Parameters

num_classes (int) – The number of classes.

Returns

The custom classification channels.

Return type

int

class mmdet.models.losses.SmoothL1Loss(beta: float = 1.0, reduction: str = 'mean', loss_weight: float = 1.0)

Smooth L1 loss.

Parameters

• beta (float, optional) – The threshold in the piecewise function. Defaults to 1.0.

• reduction (str, optional) – The method to reduce the loss. Options are “none”, “mean” and “sum”. Defaults to “mean”.

• loss_weight (float, optional) – The weight of loss.

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None, **kwargs) → Tensor

Forward function.

Parameters

• pred (Tensor) – The prediction.

• target (Tensor) – The learning target of the prediction.

• weight (Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Defaults to None.

Returns

Calculated loss

Return type

Tensor

class mmdet.models.losses.TripletLoss(margin: float = 0.3, loss_weight: float = 1.0, hard_mining=True)

Triplet loss with hard positive/negative mining.

Reference:

Hermans et al. In Defense of the Triplet Loss for

Person Re-Identification. arXiv:1703.07737.

Imported from `<https://github.com/KaiyangZhou/deep-person-reid/blob/

master/torchreid/losses/hard_mine_triplet_loss.py>`_.

Parameters

• margin (float, optional) – Margin for triplet loss. Defaults to 0.3.

• loss_weight (float, optional) – Weight of the loss. Defaults to 1.0.

• hard_mining (bool, optional) – Whether to perform hard mining. Defaults to True.

forward(inputs: Tensor, targets: LongTensor) → Tensor

Parameters

• inputs (torch.Tensor) – feature matrix with shape (batch_size, feat_dim).

• targets (torch.LongTensor) – ground truth labels with shape (num_classes).

Returns

triplet loss.

Return type

torch.Tensor

hard_mining_triplet_loss_forward(inputs: Tensor, targets: LongTensor) → Tensor

Parameters

• inputs (torch.Tensor) – feature matrix with shape (batch_size, feat_dim).

• targets (torch.LongTensor) – ground truth labels with shape (batch_size).

Returns

triplet loss with hard mining.

Return type

torch.Tensor

class mmdet.models.losses.VarifocalLoss(use_sigmoid: bool = True, alpha: float = 0.75, gamma: float = 2.0, iou_weighted: bool = True, reduction: str = 'mean', loss_weight: float = 1.0)

forward(pred: Tensor, target: Tensor, weight: Optional[Tensor] = None, avg_factor: Optional[int] = None, reduction_override: Optional[str] = None) → Tensor

Forward function.

Parameters

• pred (Tensor) – The prediction with shape (N, C), C is the number of classes.

• target (Tensor) – The learning target of the iou-aware classification score with shape (N, C), C is the number of classes.

• weight (Tensor, optional) – The weight of loss for each prediction. Defaults to None.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”.

Returns

The calculated loss

Return type

Tensor

mmdet.models.losses.accuracy(pred, target, topk=1, thresh=None)

Calculate accuracy according to the prediction and target.

Parameters

• pred (torch.Tensor) – The model prediction, shape (N, num_class)

• target (torch.Tensor) – The target of each prediction, shape (N, )

• topk (int | tuple[int], optional) – If the predictions in topk matches the target, the predictions will be regarded as correct ones. Defaults to 1.

• thresh (float, optional) – If not None, predictions with scores under this threshold are considered incorrect. Default to None.

Returns

If the input topk is a single integer,

the function will return a single float as accuracy. If topk is a tuple containing multiple integers, the function will return a tuple containing accuracies of each topk number.

Return type

float | tuple[float]

mmdet.models.losses.balanced_l1_loss(pred, target, beta=1.0, alpha=0.5, gamma=1.5, reduction='mean')

Calculate balanced L1 loss.

Please see the Libra R-CNN

Parameters

• pred (torch.Tensor) – The prediction with shape (N, 4).

• target (torch.Tensor) – The learning target of the prediction with shape (N, 4).

• beta (float) – The loss is a piecewise function of prediction and target and beta serves as a threshold for the difference between the prediction and target. Defaults to 1.0.

• alpha (float) – The denominator alpha in the balanced L1 loss. Defaults to 0.5.

• gamma (float) – The gamma in the balanced L1 loss. Defaults to 1.5.

• reduction (str, optional) – The method that reduces the loss to a scalar. Options are “none”, “mean” and “sum”.

Returns

The calculated loss

Return type

torch.Tensor

mmdet.models.losses.binary_cross_entropy(pred, label, weight=None, reduction='mean', avg_factor=None, class_weight=None, ignore_index=-100, avg_non_ignore=False)

Calculate the binary CrossEntropy loss.

Parameters

• pred (torch.Tensor) – The prediction with shape (N, 1) or (N, ). When the shape of pred is (N, 1), label will be expanded to one-hot format, and when the shape of pred is (N, ), label will not be expanded to one-hot format.

• label (torch.Tensor) – The learning label of the prediction, with shape (N, ).

• weight (torch.Tensor, optional) – Sample-wise loss weight.

• reduction (str, optional) – The method used to reduce the loss. Options are “none”, “mean” and “sum”.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• class_weight (list[float], optional) – The weight for each class.

• ignore_index (int | None) – The label index to be ignored. If None, it will be set to default value. Default: -100.

• avg_non_ignore (bool) – The flag decides to whether the loss is only averaged over non-ignored targets. Default: False.

Returns

The calculated loss.

Return type

torch.Tensor

mmdet.models.losses.bounded_iou_loss(pred: Tensor, target: Tensor, beta: float = 0.2, eps: float = 0.001) → Tensor

BIoULoss.

This is an implementation of paper Improving Object Localization with Fitness NMS and Bounded IoU Loss..

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – Corresponding gt bboxes, shape (n, 4).

• beta (float, optional) – Beta parameter in smoothl1.

• eps (float, optional) – Epsilon to avoid NaN values.

Returns

Loss tensor.

Return type

Tensor

mmdet.models.losses.carl_loss(cls_score: Tensor, labels: Tensor, bbox_pred: Tensor, bbox_targets: Tensor, loss_bbox: Module, k: float = 1, bias: float = 0.2, avg_factor: Optional[int] = None, sigmoid: bool = False, num_class: int = 80) → dict

Classification-Aware Regression Loss (CARL).

Parameters

• cls_score (Tensor) – Predicted classification scores.

• labels (Tensor) – Targets of classification.

• bbox_pred (Tensor) – Predicted bbox deltas.

• bbox_targets (Tensor) – Target of bbox regression.

• loss_bbox (func) – Regression loss func of the head.

• bbox_coder (obj) – BBox coder of the head.

• k (float) – Power of the non-linear mapping. Defaults to 1.

• bias (float) – Shift of the non-linear mapping. Defaults to 0.2.

• avg_factor (int, optional) – Average factor used in regression loss.

• sigmoid (bool) – Activation of the classification score.

• num_class (int) – Number of classes, defaults to 80.

Returns

CARL loss dict.

Return type

dict

mmdet.models.losses.cross_entropy(pred, label, weight=None, reduction='mean', avg_factor=None, class_weight=None, ignore_index=-100, avg_non_ignore=False)

Calculate the CrossEntropy loss.

Parameters

• pred (torch.Tensor) – The prediction with shape (N, C), C is the number of classes.

• label (torch.Tensor) – The learning label of the prediction.

• weight (torch.Tensor, optional) – Sample-wise loss weight.

• reduction (str, optional) – The method used to reduce the loss.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• class_weight (list[float], optional) – The weight for each class.

• ignore_index (int | None) – The label index to be ignored. If None, it will be set to default value. Default: -100.

• avg_non_ignore (bool) – The flag decides to whether the loss is only averaged over non-ignored targets. Default: False.

Returns

The calculated loss

Return type

torch.Tensor

mmdet.models.losses.iou_loss(pred: Tensor, target: Tensor, linear: bool = False, mode: str = 'log', eps: float = 1e-06) → Tensor

IoU loss.

Computing the IoU loss between a set of predicted bboxes and target bboxes. The loss is calculated as negative log of IoU.

Parameters

• pred (Tensor) – Predicted bboxes of format (x1, y1, x2, y2), shape (n, 4).

• target (Tensor) – Corresponding gt bboxes, shape (n, 4).

• linear (bool, optional) – If True, use linear scale of loss instead of log scale. Default: False.

• mode (str) – Loss scaling mode, including “linear”, “square”, and “log”. Default: ‘log’

• eps (float) – Epsilon to avoid log(0).

Returns

Loss tensor.

Return type

Tensor

mmdet.models.losses.isr_p(cls_score: Tensor, bbox_pred: Tensor, bbox_targets: Tuple[Tensor], rois: Tensor, sampling_results: List[SamplingResult], loss_cls: Module, bbox_coder: BaseBBoxCoder, k: float = 2, bias: float = 0, num_class: int = 80) → tuple

Importance-based Sample Reweighting (ISR_P), positive part.

Parameters

• cls_score (Tensor) – Predicted classification scores.

• bbox_pred (Tensor) – Predicted bbox deltas.

• bbox_targets (tuple[Tensor]) – A tuple of bbox targets, the are labels, label_weights, bbox_targets, bbox_weights, respectively.

• rois (Tensor) – Anchors (single_stage) in shape (n, 4) or RoIs (two_stage) in shape (n, 5).

• sampling_results (SamplingResult) – Sampling results.

• loss_cls (nn.Module) – Classification loss func of the head.

• bbox_coder (BaseBBoxCoder) – BBox coder of the head.

• k (float) – Power of the non-linear mapping. Defaults to 2.

• bias (float) – Shift of the non-linear mapping. Defaults to 0.

• num_class (int) – Number of classes, defaults to 80.

Returns

labels, imp_based_label_weights, bbox_targets,

bbox_target_weights

Return type

tuple([Tensor])

mmdet.models.losses.l1_loss(pred: Tensor, target: Tensor) → Tensor

L1 loss.

Parameters

• pred (Tensor) – The prediction.

• target (Tensor) – The learning target of the prediction.

Returns

Calculated loss

Return type

Tensor

mmdet.models.losses.mask_cross_entropy(pred, target, label, reduction='mean', avg_factor=None, class_weight=None, ignore_index=None, **kwargs)

Calculate the CrossEntropy loss for masks.

Parameters

• pred (torch.Tensor) – The prediction with shape (N, C, *), C is the number of classes. The trailing * indicates arbitrary shape.

• target (torch.Tensor) – The learning label of the prediction.

• label (torch.Tensor) – label indicates the class label of the mask corresponding object. This will be used to select the mask in the of the class which the object belongs to when the mask prediction if not class-agnostic.

• reduction (str, optional) – The method used to reduce the loss. Options are “none”, “mean” and “sum”.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

• class_weight (list[float], optional) – The weight for each class.

• ignore_index (None) – Placeholder, to be consistent with other loss. Default: None.

Returns

The calculated loss

Return type

torch.Tensor

Example

>>> N, C = 3, 11
>>> H, W = 2, 2
>>> pred = torch.randn(N, C, H, W) * 1000
>>> target = torch.rand(N, H, W)
>>> label = torch.randint(0, C, size=(N,))
>>> reduction = 'mean'
>>> avg_factor = None
>>> class_weights = None
>>> loss = mask_cross_entropy(pred, target, label, reduction,
>>>                           avg_factor, class_weights)
>>> assert loss.shape == (1,)

mmdet.models.losses.mse_loss(pred: Tensor, target: Tensor) → Tensor

A Wrapper of MSE loss. :param pred: The prediction. :type pred: Tensor :param target: The learning target of the prediction. :type target: Tensor

Returns

loss Tensor

Return type

Tensor

mmdet.models.losses.reduce_loss(loss: Tensor, reduction: str) → Tensor

Reduce loss as specified.

Parameters

• loss (Tensor) – Elementwise loss tensor.

• reduction (str) – Options are “none”, “mean” and “sum”.

Returns

Reduced loss tensor.

Return type

Tensor

mmdet.models.losses.sigmoid_focal_loss(pred, target, weight=None, gamma=2.0, alpha=0.25, reduction='mean', avg_factor=None)

A wrapper of cuda version Focal Loss.

Parameters

• pred (torch.Tensor) – The prediction with shape (N, C), C is the number of classes.

• target (torch.Tensor) – The learning label of the prediction.

• weight (torch.Tensor, optional) – Sample-wise loss weight.

• gamma (float, optional) – The gamma for calculating the modulating factor. Defaults to 2.0.

• alpha (float, optional) – A balanced form for Focal Loss. Defaults to 0.25.

• reduction (str, optional) – The method used to reduce the loss into a scalar. Defaults to ‘mean’. Options are “none”, “mean” and “sum”.

• avg_factor (int, optional) – Average factor that is used to average the loss. Defaults to None.

mmdet.models.losses.smooth_l1_loss(pred: Tensor, target: Tensor, beta: float = 1.0) → Tensor

Smooth L1 loss.

Parameters

• pred (Tensor) – The prediction.

• target (Tensor) – The learning target of the prediction.

• beta (float, optional) – The threshold in the piecewise function. Defaults to 1.0.

Returns

Calculated loss

Return type

Tensor

mmdet.models.losses.weight_reduce_loss(loss: Tensor, weight: Optional[Tensor] = None, reduction: str = 'mean', avg_factor: Optional[float] = None) → Tensor

Apply element-wise weight and reduce loss.

Parameters

• loss (Tensor) – Element-wise loss.

• weight (Optional[Tensor], optional) – Element-wise weights. Defaults to None.

• reduction (str, optional) – Same as built-in losses of PyTorch. Defaults to ‘mean’.

• avg_factor (Optional[float], optional) – Average factor when computing the mean of losses. Defaults to None.

Returns

Processed loss values.

Return type

Tensor

mmdet.models.losses.weighted_loss(loss_func: Callable) → Callable

Create a weighted version of a given loss function.

To use this decorator, the loss function must have the signature like loss_func(pred, target, **kwargs). The function only needs to compute element-wise loss without any reduction. This decorator will add weight and reduction arguments to the function. The decorated function will have the signature like loss_func(pred, target, weight=None, reduction=’mean’, avg_factor=None, **kwargs).

Example

>>> import torch
>>> @weighted_loss
>>> def l1_loss(pred, target):
>>>     return (pred - target).abs()

>>> pred = torch.Tensor([0, 2, 3])
>>> target = torch.Tensor([1, 1, 1])
>>> weight = torch.Tensor([1, 0, 1])

>>> l1_loss(pred, target)
tensor(1.3333)
>>> l1_loss(pred, target, weight)
tensor(1.)
>>> l1_loss(pred, target, reduction='none')
tensor([1., 1., 2.])
>>> l1_loss(pred, target, weight, avg_factor=2)
tensor(1.5000)

necks

class mmdet.models.necks.BFP(Balanced Feature Pyramids)

BFP takes multi-level features as inputs and gather them into a single one, then refine the gathered feature and scatter the refined results to multi-level features. This module is used in Libra R-CNN (CVPR 2019), see the paper Libra R-CNN: Towards Balanced Learning for Object Detection for details.

Parameters

• in_channels (int) – Number of input channels (feature maps of all levels should have the same channels).

• num_levels (int) – Number of input feature levels.

• refine_level (int) – Index of integration and refine level of BSF in multi-level features from bottom to top.

• refine_type (str) – Type of the refine op, currently support [None, ‘conv’, ‘non_local’].

• conv_cfg (ConfigDict or dict, optional) – The config dict for convolution layers.

• norm_cfg (ConfigDict or dict, optional) – The config dict for normalization layers.

:param init_cfg (ConfigDict or dict or list[ConfigDict or: dict], optional): Initialization config dict.

forward(inputs: Tuple[Tensor]) → Tuple[Tensor]

Forward function.

class mmdet.models.necks.CLRFPN(in_channels, out_channels, num_outs, start_level=0, end_level=-1, add_extra_convs=False, extra_convs_on_inputs=True, relu_before_extra_convs=False, no_norm_on_lateral=False, conv_cfg=None, norm_cfg=None, attention=False, act_cfg=None, upsample_cfg={'mode': 'nearest'}, init_cfg={'distribution': 'uniform', 'layer': 'Conv2d', 'type': 'Xavier'}, cfg=None)

forward(inputs)

Forward function.

class mmdet.models.necks.CSPNeXtPAFPN(in_channels: Sequence[int], out_channels: int, num_csp_blocks: int = 3, use_depthwise: bool = False, expand_ratio: float = 0.5, upsample_cfg: Union[ConfigDict, dict] = {'mode': 'nearest', 'scale_factor': 2}, conv_cfg: Optional[bool] = None, norm_cfg: Union[ConfigDict, dict] = {'eps': 0.001, 'momentum': 0.03, 'type': 'BN'}, act_cfg: Union[ConfigDict, dict] = {'type': 'Swish'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = {'a': 2.23606797749979, 'distribution': 'uniform', 'layer': 'Conv2d', 'mode': 'fan_in', 'nonlinearity': 'leaky_relu', 'type': 'Kaiming'})

Path Aggregation Network with CSPNeXt blocks.

Parameters

• in_channels (Sequence[int]) – Number of input channels per scale.

• out_channels (int) – Number of output channels (used at each scale)

• num_csp_blocks (int) – Number of bottlenecks in CSPLayer. Defaults to 3.

• use_depthwise (bool) – Whether to use depthwise separable convolution in blocks. Defaults to False.

• expand_ratio (float) – Ratio to adjust the number of channels of the hidden layer. Default: 0.5

• upsample_cfg (dict) – Config dict for interpolate layer. Default: dict(scale_factor=2, mode=’nearest’)

• conv_cfg (dict, optional) – Config dict for convolution layer. Default: None, which means using conv2d.

• norm_cfg (dict) – Config dict for normalization layer. Default: dict(type=’BN’)

• act_cfg (dict) – Config dict for activation layer. Default: dict(type=’Swish’)

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None.

forward(inputs: Tuple[Tensor, ...]) → Tuple[Tensor, ...]

Parameters

inputs (tuple[Tensor]) – input features.

Returns

YOLOXPAFPN features.

Return type

tuple[Tensor]

class mmdet.models.necks.CTResNetNeck(in_channels: int, num_deconv_filters: Tuple[int, ...], num_deconv_kernels: Tuple[int, ...], use_dcn: bool = True, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

The neck used in CenterNet for object classification and box regression.

Parameters

• in_channels (int) – Number of input channels.

• num_deconv_filters (tuple[int]) – Number of filters per stage.

• num_deconv_kernels (tuple[int]) – Number of kernels per stage.

• use_dcn (bool) – If True, use DCNv2. Defaults to True.

:param init_cfg (ConfigDict or dict or list[dict] or: list[ConfigDict], optional): Initialization

config dict.

forward(x: Sequence[Tensor]) → Tuple[Tensor]

Model forward.

init_weights() → None

Initialize the parameters.

class mmdet.models.necks.ChannelMapper(in_channels: List[int], out_channels: int, kernel_size: int = 3, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, act_cfg: Optional[Union[ConfigDict, dict]] = {'type': 'ReLU'}, bias: Union[bool, str] = 'auto', num_outs: Optional[int] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = {'distribution': 'uniform', 'layer': 'Conv2d', 'type': 'Xavier'})

Channel Mapper to reduce/increase channels of backbone features.

This is used to reduce/increase channels of backbone features.

Parameters

• in_channels (List[int]) – Number of input channels per scale.

• out_channels (int) – Number of output channels (used at each scale).

• kernel_size (int, optional) – kernel_size for reducing channels (used at each scale). Default: 3.

• conv_cfg (ConfigDict or dict, optional) – Config dict for convolution layer. Default: None.

• norm_cfg (ConfigDict or dict, optional) – Config dict for normalization layer. Default: None.

• act_cfg (ConfigDict or dict, optional) – Config dict for activation layer in ConvModule. Default: dict(type=’ReLU’).

• bias (bool | str) – If specified as auto, it will be decided by the norm_cfg. Bias will be set as True if norm_cfg is None, otherwise False. Default: “auto”.

• num_outs (int, optional) – Number of output feature maps. There would be extra_convs when num_outs larger than the length of in_channels.

:param init_cfg (ConfigDict or dict or list[ConfigDict or dict]: optional): Initialization config dict. :param : optional): Initialization config dict.

Example

>>> import torch
>>> in_channels = [2, 3, 5, 7]
>>> scales = [340, 170, 84, 43]
>>> inputs = [torch.rand(1, c, s, s)
...           for c, s in zip(in_channels, scales)]
>>> self = ChannelMapper(in_channels, 11, 3).eval()
>>> outputs = self.forward(inputs)
>>> for i in range(len(outputs)):
...     print(f'outputs[{i}].shape = {outputs[i].shape}')
outputs[0].shape = torch.Size([1, 11, 340, 340])
outputs[1].shape = torch.Size([1, 11, 170, 170])
outputs[2].shape = torch.Size([1, 11, 84, 84])
outputs[3].shape = torch.Size([1, 11, 43, 43])

forward(inputs: Tuple[Tensor]) → Tuple[Tensor]

Forward function.

class mmdet.models.necks.DilatedEncoder(in_channels, out_channels, block_mid_channels, num_residual_blocks, block_dilations)

Dilated Encoder for YOLOF <https://arxiv.org/abs/2103.09460>`.

This module contains two types of components:

• the original FPN lateral convolution layer and fpn convolution layer,

  which are 1x1 conv + 3x3 conv

• the dilated residual block

Parameters

• in_channels (int) – The number of input channels.

• out_channels (int) – The number of output channels.

• block_mid_channels (int) – The number of middle block output channels

• num_residual_blocks (int) – The number of residual blocks.

• block_dilations (list) – The list of residual blocks dilation.

forward(feature)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.necks.DyHead(in_channels, out_channels, num_blocks=6, zero_init_offset=True, init_cfg=None)

DyHead neck consisting of multiple DyHead Blocks.

See Dynamic Head: Unifying Object Detection Heads with Attentions for details.

Parameters

• in_channels (int) – Number of input channels.

• out_channels (int) – Number of output channels.

• num_blocks (int, optional) – Number of DyHead Blocks. Default: 6.

• zero_init_offset (bool, optional) – Whether to use zero init for spatial_conv_offset. Default: True.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None.

forward(inputs)

Forward function.

class mmdet.models.necks.FPG(in_channels, out_channels, num_outs, stack_times, paths, inter_channels=None, same_down_trans=None, same_up_trans={'kernel_size': 3, 'padding': 1, 'stride': 2, 'type': 'conv'}, across_lateral_trans={'kernel_size': 1, 'type': 'conv'}, across_down_trans={'kernel_size': 3, 'type': 'conv'}, across_up_trans=None, across_skip_trans={'type': 'identity'}, output_trans={'kernel_size': 3, 'type': 'last_conv'}, start_level=0, end_level=-1, add_extra_convs=False, norm_cfg=None, skip_inds=None, init_cfg=[{'type': 'Caffe2Xavier', 'layer': 'Conv2d'}, {'type': 'Constant', 'layer': ['_BatchNorm', '_InstanceNorm', 'GroupNorm', 'LayerNorm'], 'val': 1.0}])

FPG.

Implementation of Feature Pyramid Grids (FPG). This implementation only gives the basic structure stated in the paper. But users can implement different type of transitions to fully explore the the potential power of the structure of FPG.

Parameters

• in_channels (int) – Number of input channels (feature maps of all levels should have the same channels).

• out_channels (int) – Number of output channels (used at each scale)

• num_outs (int) – Number of output scales.

• stack_times (int) – The number of times the pyramid architecture will be stacked.

• paths (list[str]) – Specify the path order of each stack level. Each element in the list should be either ‘bu’ (bottom-up) or ‘td’ (top-down).

• inter_channels (int) – Number of inter channels.

• same_up_trans (dict) – Transition that goes down at the same stage.

• same_down_trans (dict) – Transition that goes up at the same stage.

• across_lateral_trans (dict) – Across-pathway same-stage

• across_down_trans (dict) – Across-pathway bottom-up connection.

• across_up_trans (dict) – Across-pathway top-down connection.

• across_skip_trans (dict) – Across-pathway skip connection.

• output_trans (dict) – Transition that trans the output of the last stage.

• start_level (int) – Index of the start input backbone level used to build the feature pyramid. Default: 0.

• end_level (int) – Index of the end input backbone level (exclusive) to build the feature pyramid. Default: -1, which means the last level.

• add_extra_convs (bool) – It decides whether to add conv layers on top of the original feature maps. Default to False. If True, its actual mode is specified by extra_convs_on_inputs.

• norm_cfg (dict) – Config dict for normalization layer. Default: None.

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(inputs)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.necks.FPN(in_channels: List[int], out_channels: int, num_outs: int, start_level: int = 0, end_level: int = -1, add_extra_convs: Union[bool, str] = False, relu_before_extra_convs: bool = False, no_norm_on_lateral: bool = False, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, act_cfg: Optional[Union[ConfigDict, dict]] = None, upsample_cfg: Union[ConfigDict, dict] = {'mode': 'nearest'}, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'distribution': 'uniform', 'layer': 'Conv2d', 'type': 'Xavier'})

Feature Pyramid Network.

This is an implementation of paper Feature Pyramid Networks for Object Detection.

Parameters

• in_channels (list[int]) – Number of input channels per scale.

• out_channels (int) – Number of output channels (used at each scale).

• num_outs (int) – Number of output scales.

• start_level (int) – Index of the start input backbone level used to build the feature pyramid. Defaults to 0.

• end_level (int) – Index of the end input backbone level (exclusive) to build the feature pyramid. Defaults to -1, which means the last level.

• add_extra_convs (bool | str) –

  If bool, it decides whether to add conv layers on top of the original feature maps. Defaults to False. If True, it is equivalent to add_extra_convs=’on_input’. If str, it specifies the source feature map of the extra convs. Only the following options are allowed

  • ’on_input’: Last feat map of neck inputs (i.e. backbone feature).

  • ’on_lateral’: Last feature map after lateral convs.

  • ’on_output’: The last output feature map after fpn convs.

• relu_before_extra_convs (bool) – Whether to apply relu before the extra conv. Defaults to False.

• no_norm_on_lateral (bool) – Whether to apply norm on lateral. Defaults to False.

• conv_cfg (ConfigDict or dict, optional) – Config dict for convolution layer. Defaults to None.

• norm_cfg (ConfigDict or dict, optional) – Config dict for normalization layer. Defaults to None.

• act_cfg (ConfigDict or dict, optional) – Config dict for activation layer in ConvModule. Defaults to None.

• upsample_cfg (ConfigDict or dict, optional) – Config dict for interpolate layer. Defaults to dict(mode=’nearest’).

:param init_cfg (ConfigDict or dict or list[ConfigDict or : dict]): Initialization config dict.

Example

>>> import torch
>>> in_channels = [2, 3, 5, 7]
>>> scales = [340, 170, 84, 43]
>>> inputs = [torch.rand(1, c, s, s)
...           for c, s in zip(in_channels, scales)]
>>> self = FPN(in_channels, 11, len(in_channels)).eval()
>>> outputs = self.forward(inputs)
>>> for i in range(len(outputs)):
...     print(f'outputs[{i}].shape = {outputs[i].shape}')
outputs[0].shape = torch.Size([1, 11, 340, 340])
outputs[1].shape = torch.Size([1, 11, 170, 170])
outputs[2].shape = torch.Size([1, 11, 84, 84])
outputs[3].shape = torch.Size([1, 11, 43, 43])

forward(inputs: Tuple[Tensor]) → tuple

Forward function.

Parameters

inputs (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Feature maps, each is a 4D-tensor.

Return type

tuple

class mmdet.models.necks.FPN_CARAFE(in_channels, out_channels, num_outs, start_level=0, end_level=-1, norm_cfg=None, act_cfg=None, order=('conv', 'norm', 'act'), upsample_cfg={'encoder_dilation': 1, 'encoder_kernel': 3, 'type': 'carafe', 'up_group': 1, 'up_kernel': 5}, init_cfg=None)

FPN_CARAFE is a more flexible implementation of FPN. It allows more choice for upsample methods during the top-down pathway.

It can reproduce the performance of ICCV 2019 paper CARAFE: Content-Aware ReAssembly of FEatures Please refer to https://arxiv.org/abs/1905.02188 for more details.

Parameters

• in_channels (list[int]) – Number of channels for each input feature map.

• out_channels (int) – Output channels of feature pyramids.

• num_outs (int) – Number of output stages.

• start_level (int) – Start level of feature pyramids. (Default: 0)

• end_level (int) – End level of feature pyramids. (Default: -1 indicates the last level).

• norm_cfg (dict) – Dictionary to construct and config norm layer.

• activate (str) – Type of activation function in ConvModule (Default: None indicates w/o activation).

• order (dict) – Order of components in ConvModule.

• upsample (str) – Type of upsample layer.

• upsample_cfg (dict) – Dictionary to construct and config upsample layer.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

forward(inputs)

Forward function.

init_weights()

Initialize the weights of module.

slice_as(src, dst)

Slice src as dst

Note

src should have the same or larger size than dst.

Parameters

• src (torch.Tensor) – Tensors to be sliced.

• dst (torch.Tensor) – src will be sliced to have the same size as dst.

Returns

Sliced tensor.

Return type

torch.Tensor

tensor_add(a, b)

Add tensors a and b that might have different sizes.

class mmdet.models.necks.FPN_DropBlock(*args, plugin: Optional[dict] = {'block_size': 3, 'drop_prob': 0.3, 'type': 'DropBlock', 'warmup_iters': 0}, **kwargs)

forward(inputs: Tuple[Tensor]) → tuple

Forward function.

Parameters

inputs (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Feature maps, each is a 4D-tensor.

Return type

tuple

class mmdet.models.necks.HRFPN(High Resolution Feature Pyramids)

paper: High-Resolution Representations for Labeling Pixels and Regions.

Parameters

• in_channels (list) – number of channels for each branch.

• out_channels (int) – output channels of feature pyramids.

• num_outs (int) – number of output stages.

• pooling_type (str) – pooling for generating feature pyramids from {MAX, AVG}.

• conv_cfg (dict) – dictionary to construct and config conv layer.

• norm_cfg (dict) – dictionary to construct and config norm layer.

• with_cp (bool) – Use checkpoint or not. Using checkpoint will save some memory while slowing down the training speed.

• stride (int) – stride of 3x3 convolutional layers

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(inputs)

Forward function.

class mmdet.models.necks.NASFCOS_FPN(in_channels, out_channels, num_outs, start_level=1, end_level=-1, add_extra_convs=False, conv_cfg=None, norm_cfg=None, init_cfg=None)

FPN structure in NASFPN.

Implementation of paper NAS-FCOS: Fast Neural Architecture Search for Object Detection

Parameters

• in_channels (List[int]) – Number of input channels per scale.

• out_channels (int) – Number of output channels (used at each scale)

• num_outs (int) – Number of output scales.

• start_level (int) – Index of the start input backbone level used to build the feature pyramid. Default: 0.

• end_level (int) – Index of the end input backbone level (exclusive) to build the feature pyramid. Default: -1, which means the last level.

• add_extra_convs (bool) – It decides whether to add conv layers on top of the original feature maps. Default to False. If True, its actual mode is specified by extra_convs_on_inputs.

• conv_cfg (dict) – dictionary to construct and config conv layer.

• norm_cfg (dict) – dictionary to construct and config norm layer.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

forward(inputs)

Forward function.

init_weights()

Initialize the weights of module.

class mmdet.models.necks.NASFPN(in_channels: List[int], out_channels: int, num_outs: int, stack_times: int, start_level: int = 0, end_level: int = -1, norm_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Conv2d', 'type': 'Caffe2Xavier'})

NAS-FPN.

Implementation of NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection

Parameters

• in_channels (List[int]) – Number of input channels per scale.

• out_channels (int) – Number of output channels (used at each scale)

• num_outs (int) – Number of output scales.

• stack_times (int) – The number of times the pyramid architecture will be stacked.

• start_level (int) – Index of the start input backbone level used to build the feature pyramid. Defaults to 0.

• end_level (int) – Index of the end input backbone level (exclusive) to build the feature pyramid. Defaults to -1, which means the last level.

• norm_cfg (ConfigDict or dict, optional) – Config dict for normalization layer. Defaults to None.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict]) – Initialization config dict.

forward(inputs: Tuple[Tensor]) → tuple

Forward function.

Args:

inputs (tuple[Tensor]): Features from the upstream network, each

is a 4D-tensor.

Returns

Feature maps, each is a 4D-tensor.

Return type

tuple

class mmdet.models.necks.PAFPN(in_channels, out_channels, num_outs, start_level=0, end_level=-1, add_extra_convs=False, relu_before_extra_convs=False, no_norm_on_lateral=False, conv_cfg=None, norm_cfg=None, act_cfg=None, init_cfg={'distribution': 'uniform', 'layer': 'Conv2d', 'type': 'Xavier'})

Path Aggregation Network for Instance Segmentation.

This is an implementation of the PAFPN in Path Aggregation Network.

Parameters

• in_channels (List[int]) – Number of input channels per scale.

• out_channels (int) – Number of output channels (used at each scale)

• num_outs (int) – Number of output scales.

• start_level (int) – Index of the start input backbone level used to build the feature pyramid. Default: 0.

• end_level (int) – Index of the end input backbone level (exclusive) to build the feature pyramid. Default: -1, which means the last level.

• add_extra_convs (bool | str) –

  If bool, it decides whether to add conv layers on top of the original feature maps. Default to False. If True, it is equivalent to add_extra_convs=’on_input’. If str, it specifies the source feature map of the extra convs. Only the following options are allowed

  • ’on_input’: Last feat map of neck inputs (i.e. backbone feature).

  • ’on_lateral’: Last feature map after lateral convs.

  • ’on_output’: The last output feature map after fpn convs.

• relu_before_extra_convs (bool) – Whether to apply relu before the extra conv. Default: False.

• no_norm_on_lateral (bool) – Whether to apply norm on lateral. Default: False.

• conv_cfg (dict) – Config dict for convolution layer. Default: None.

• norm_cfg (dict) – Config dict for normalization layer. Default: None.

• act_cfg (str) – Config dict for activation layer in ConvModule. Default: None.

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(inputs)

Forward function.

class mmdet.models.necks.RFP(Recursive Feature Pyramid)

This is an implementation of RFP in DetectoRS. Different from standard FPN, the input of RFP should be multi level features along with origin input image of backbone.

Parameters

• rfp_steps (int) – Number of unrolled steps of RFP.

• rfp_backbone (dict) – Configuration of the backbone for RFP.

• aspp_out_channels (int) – Number of output channels of ASPP module.

• aspp_dilations (tuple[int]) – Dilation rates of four branches. Default: (1, 3, 6, 1)

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

forward(inputs)

Forward function.

Parameters

inputs (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

Feature maps, each is a 4D-tensor.

Return type

tuple

init_weights()

Initialize the weights.

class mmdet.models.necks.SSDNeck(in_channels, out_channels, level_strides, level_paddings, l2_norm_scale=20.0, last_kernel_size=3, use_depthwise=False, conv_cfg=None, norm_cfg=None, act_cfg={'type': 'ReLU'}, init_cfg=[{'type': 'Xavier', 'distribution': 'uniform', 'layer': 'Conv2d'}, {'type': 'Constant', 'val': 1, 'layer': 'BatchNorm2d'}])

Extra layers of SSD backbone to generate multi-scale feature maps.

Parameters

• in_channels (Sequence[int]) – Number of input channels per scale.

• out_channels (Sequence[int]) – Number of output channels per scale.

• level_strides (Sequence[int]) – Stride of 3x3 conv per level.

• level_paddings (Sequence[int]) – Padding size of 3x3 conv per level.

• l2_norm_scale (float|None) – L2 normalization layer init scale. If None, not use L2 normalization on the first input feature.

• last_kernel_size (int) – Kernel size of the last conv layer. Default: 3.

• use_depthwise (bool) – Whether to use DepthwiseSeparableConv. Default: False.

• conv_cfg (dict) – Config dict for convolution layer. Default: None.

• norm_cfg (dict) – Dictionary to construct and config norm layer. Default: None.

• act_cfg (dict) – Config dict for activation layer. Default: dict(type=’ReLU’).

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(inputs)

Forward function.

class mmdet.models.necks.SSH(num_scales: int, in_channels: List[int], out_channels: List[int], conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'type': 'BN'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = {'distribution': 'uniform', 'layer': 'Conv2d', 'type': 'Xavier'})

SSH Neck used in `SSH: Single Stage Headless Face Detector.

<https://arxiv.org/pdf/1708.03979.pdf>`_.

Parameters

• num_scales (int) – The number of scales / stages.

• in_channels (list[int]) – The number of input channels per scale.

• out_channels (list[int]) – The number of output channels per scale.

• conv_cfg (ConfigDict or dict, optional) – Config dict for convolution layer. Defaults to None.

• norm_cfg (ConfigDict or dict) – Config dict for normalization layer. Defaults to dict(type=’BN’).

:param init_cfg (ConfigDict or list[ConfigDict] or dict or: list[dict], optional): Initialization config dict.

Example

>>> import torch
>>> in_channels = [8, 16, 32, 64]
>>> out_channels = [16, 32, 64, 128]
>>> scales = [340, 170, 84, 43]
>>> inputs = [torch.rand(1, c, s, s)
...           for c, s in zip(in_channels, scales)]
>>> self = SSH(num_scales=4, in_channels=in_channels,
...           out_channels=out_channels)
>>> outputs = self.forward(inputs)
>>> for i in range(len(outputs)):
...     print(f'outputs[{i}].shape = {outputs[i].shape}')
outputs[0].shape = torch.Size([1, 16, 340, 340])
outputs[1].shape = torch.Size([1, 32, 170, 170])
outputs[2].shape = torch.Size([1, 64, 84, 84])
outputs[3].shape = torch.Size([1, 128, 43, 43])

forward(inputs: Tuple[Tensor]) → tuple

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.necks.YOLOV3Neck(num_scales: int, in_channels: List[int], out_channels: List[int], conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'requires_grad': True, 'type': 'BN'}, act_cfg: Union[ConfigDict, dict] = {'negative_slope': 0.1, 'type': 'LeakyReLU'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

The neck of YOLOV3.

It can be treated as a simplified version of FPN. It will take the result from Darknet backbone and do some upsampling and concatenation. It will finally output the detection result.

Note

The input feats should be from top to bottom.

i.e., from high-lvl to low-lvl

But YOLOV3Neck will process them in reversed order.

i.e., from bottom (high-lvl) to top (low-lvl)

Parameters

• num_scales (int) – The number of scales / stages.

• in_channels (List[int]) – The number of input channels per scale.

• out_channels (List[int]) – The number of output channels per scale.

• conv_cfg (dict, optional) – Config dict for convolution layer. Default: None.

• norm_cfg (dict, optional) – Dictionary to construct and config norm layer. Default: dict(type=’BN’, requires_grad=True)

• act_cfg (dict, optional) – Config dict for activation layer. Default: dict(type=’LeakyReLU’, negative_slope=0.1).

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None

forward(feats=typing.Tuple[torch.Tensor]) → Tuple[Tensor]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mmdet.models.necks.YOLOXPAFPN(in_channels, out_channels, num_csp_blocks=3, use_depthwise=False, upsample_cfg={'mode': 'nearest', 'scale_factor': 2}, conv_cfg=None, norm_cfg={'eps': 0.001, 'momentum': 0.03, 'type': 'BN'}, act_cfg={'type': 'Swish'}, init_cfg={'a': 2.23606797749979, 'distribution': 'uniform', 'layer': 'Conv2d', 'mode': 'fan_in', 'nonlinearity': 'leaky_relu', 'type': 'Kaiming'})

Path Aggregation Network used in YOLOX.

Parameters

• in_channels (List[int]) – Number of input channels per scale.

• out_channels (int) – Number of output channels (used at each scale)

• num_csp_blocks (int) – Number of bottlenecks in CSPLayer. Default: 3

• use_depthwise (bool) – Whether to depthwise separable convolution in blocks. Default: False

• upsample_cfg (dict) – Config dict for interpolate layer. Default: dict(scale_factor=2, mode=’nearest’)

• conv_cfg (dict, optional) – Config dict for convolution layer. Default: None, which means using conv2d.

• norm_cfg (dict) – Config dict for normalization layer. Default: dict(type=’BN’)

• act_cfg (dict) – Config dict for activation layer. Default: dict(type=’Swish’)

• init_cfg (dict or list[dict], optional) – Initialization config dict. Default: None.

forward(inputs)

Parameters

inputs (tuple[Tensor]) – input features.

Returns

YOLOXPAFPN features.

Return type

tuple[Tensor]

roi_heads

class mmdet.models.roi_heads.BBoxHead(with_avg_pool: bool = False, with_cls: bool = True, with_reg: bool = True, roi_feat_size: int = 7, in_channels: int = 256, num_classes: int = 80, bbox_coder: Union[ConfigDict, dict] = {'clip_border': True, 'target_means': [0.0, 0.0, 0.0, 0.0], 'target_stds': [0.1, 0.1, 0.2, 0.2], 'type': 'DeltaXYWHBBoxCoder'}, predict_box_type: str = 'hbox', reg_class_agnostic: bool = False, reg_decoded_bbox: bool = False, reg_predictor_cfg: Union[ConfigDict, dict] = {'type': 'Linear'}, cls_predictor_cfg: Union[ConfigDict, dict] = {'type': 'Linear'}, loss_cls: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': False}, loss_bbox: Union[ConfigDict, dict] = {'beta': 1.0, 'loss_weight': 1.0, 'type': 'SmoothL1Loss'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Simplest RoI head, with only two fc layers for classification and regression respectively.

property custom_accuracy: bool

Get custom_accuracy from loss_cls.

property custom_activation: bool

Get custom_activation from loss_cls.

property custom_cls_channels: bool

Get custom_cls_channels from loss_cls.

forward(x: Tuple[Tensor]) → tuple

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of classification scores and bbox prediction.

• cls_score (Tensor): Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_pred (Tensor): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

Return type

tuple

get_targets(sampling_results: List[SamplingResult], rcnn_train_cfg: ConfigDict, concat: bool = True) → tuple

Calculate the ground truth for all samples in a batch according to the sampling_results.

Almost the same as the implementation in bbox_head, we passed additional parameters pos_inds_list and neg_inds_list to _get_targets_single function.

Parameters

• (List[obj (sampling_results) – SamplingResult]): Assign results of all images in a batch after sampling.

• (obj (rcnn_train_cfg) – ConfigDict): train_cfg of RCNN.

• concat (bool) – Whether to concatenate the results of all the images in a single batch.

Returns

Ground truth for proposals in a single image. Containing the following list of Tensors:

• labels (list[Tensor],Tensor): Gt_labels for all

  proposals in a batch, each tensor in list has shape (num_proposals,) when concat=False, otherwise just a single tensor has shape (num_all_proposals,).

• label_weights (list[Tensor]): Labels_weights for

  all proposals in a batch, each tensor in list has shape (num_proposals,) when concat=False, otherwise just a single tensor has shape (num_all_proposals,).

• bbox_targets (list[Tensor],Tensor): Regression target

  for all proposals in a batch, each tensor in list has shape (num_proposals, 4) when concat=False, otherwise just a single tensor has shape (num_all_proposals, 4), the last dimension 4 represents [tl_x, tl_y, br_x, br_y].

• bbox_weights (list[tensor],Tensor): Regression weights for

  all proposals in a batch, each tensor in list has shape (num_proposals, 4) when concat=False, otherwise just a single tensor has shape (num_all_proposals, 4).

Return type

Tuple[Tensor]

loss(cls_score: Tensor, bbox_pred: Tensor, rois: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tensor, bbox_weights: Tensor, reduction_override: Optional[str] = None) → dict

Calculate the loss based on the network predictions and targets.

Parameters

• cls_score (Tensor) – Classification prediction results of all class, has shape (batch_size * num_proposals_single_image, num_classes)

• bbox_pred (Tensor) – Regression prediction results, has shape (batch_size * num_proposals_single_image, 4), the last dimension 4 represents [tl_x, tl_y, br_x, br_y].

• rois (Tensor) – RoIs with the shape (batch_size * num_proposals_single_image, 5) where the first column indicates batch id of each RoI.

• labels (Tensor) – Gt_labels for all proposals in a batch, has shape (batch_size * num_proposals_single_image, ).

• label_weights (Tensor) – Labels_weights for all proposals in a batch, has shape (batch_size * num_proposals_single_image, ).

• bbox_targets (Tensor) – Regression target for all proposals in a batch, has shape (batch_size * num_proposals_single_image, 4), the last dimension 4 represents [tl_x, tl_y, br_x, br_y].

• bbox_weights (Tensor) – Regression weights for all proposals in a batch, has shape (batch_size * num_proposals_single_image, 4).

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”. Defaults to None,

Returns

A dictionary of loss.

Return type

dict

loss_and_target(cls_score: Tensor, bbox_pred: Tensor, rois: Tensor, sampling_results: List[SamplingResult], rcnn_train_cfg: ConfigDict, concat: bool = True, reduction_override: Optional[str] = None) → dict

Calculate the loss based on the features extracted by the bbox head.

Parameters

• cls_score (Tensor) – Classification prediction results of all class, has shape (batch_size * num_proposals_single_image, num_classes)

• bbox_pred (Tensor) – Regression prediction results, has shape (batch_size * num_proposals_single_image, 4), the last dimension 4 represents [tl_x, tl_y, br_x, br_y].

• rois (Tensor) – RoIs with the shape (batch_size * num_proposals_single_image, 5) where the first column indicates batch id of each RoI.

• (List[obj (sampling_results) – SamplingResult]): Assign results of all images in a batch after sampling.

• (obj (rcnn_train_cfg) – ConfigDict): train_cfg of RCNN.

• concat (bool) – Whether to concatenate the results of all the images in a single batch. Defaults to True.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”. Defaults to None,

Returns

A dictionary of loss and targets components.

The targets are only used for cascade rcnn.

Return type

dict

predict_by_feat(rois: Tuple[Tensor], cls_scores: Tuple[Tensor], bbox_preds: Tuple[Tensor], batch_img_metas: List[dict], rcnn_test_cfg: Optional[ConfigDict] = None, rescale: bool = False) → List[InstanceData]

Transform a batch of output features extracted from the head into bbox results.

Parameters

• rois (tuple[Tensor]) – Tuple of boxes to be transformed. Each has shape (num_boxes, 5). last dimension 5 arrange as (batch_index, x1, y1, x2, y2).

• cls_scores (tuple[Tensor]) – Tuple of box scores, each has shape (num_boxes, num_classes + 1).

• bbox_preds (tuple[Tensor]) – Tuple of box energies / deltas, each has shape (num_boxes, num_classes * 4).

• batch_img_metas (list[dict]) – List of image information.

• (obj (rcnn_test_cfg) – ConfigDict, optional): test_cfg of R-CNN. Defaults to None.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Instance segmentation results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

refine_bboxes(sampling_results: Union[List[SamplingResult], List[InstanceData]], bbox_results: dict, batch_img_metas: List[dict]) → List[InstanceData]

Refine bboxes during training.

:param sampling_results (List[SamplingResult] or: List[InstanceData]): Sampling results.

SamplingResult is the real sampling results calculate from bbox_head, while InstanceData is fake sampling results, e.g., in Sparse R-CNN or QueryInst, etc.

Parameters

• bbox_results (dict) –

  Usually is a dictionary with keys:

  • cls_score (Tensor): Classification scores.

  • bbox_pred (Tensor): Box energies / deltas.

  • rois (Tensor): RoIs with the shape (n, 5) where the first column indicates batch id of each RoI.

  • bbox_targets (tuple): Ground truth for proposals in a single image. Containing the following list of Tensors: (labels, label_weights, bbox_targets, bbox_weights)

• batch_img_metas (List[dict]) – List of image information.

Returns

Refined bboxes of each image.

Return type

list[InstanceData]

Example

>>> # xdoctest: +REQUIRES(module:kwarray)
>>> import numpy as np
>>> from mmdet.models.task_modules.samplers.sampling_result             >>>     import random_boxes
>>> from mmdet.models.task_modules.samplers import SamplingResult
>>> self = BBoxHead(reg_class_agnostic=True)
>>> n_roi = 2
>>> n_img = 4
>>> scale = 512
>>> rng = np.random.RandomState(0)
... batch_img_metas = [{'img_shape': (scale, scale)}
>>>                     for _ in range(n_img)]
>>> sampling_results = [SamplingResult.random(rng=10)
...                     for _ in range(n_img)]
>>> # Create rois in the expected format
>>> roi_boxes = random_boxes(n_roi, scale=scale, rng=rng)
>>> img_ids = torch.randint(0, n_img, (n_roi,))
>>> img_ids = img_ids.float()
>>> rois = torch.cat([img_ids[:, None], roi_boxes], dim=1)
>>> # Create other args
>>> labels = torch.randint(0, 81, (scale,)).long()
>>> bbox_preds = random_boxes(n_roi, scale=scale, rng=rng)
>>> cls_score = torch.randn((scale, 81))
... # For each image, pretend random positive boxes are gts
>>> bbox_targets = (labels, None, None, None)
... bbox_results = dict(rois=rois, bbox_pred=bbox_preds,
...                     cls_score=cls_score,
...                     bbox_targets=bbox_targets)
>>> bboxes_list = self.refine_bboxes(sampling_results,
...                                  bbox_results,
...                                  batch_img_metas)
>>> print(bboxes_list)

regress_by_class(priors: Tensor, label: Tensor, bbox_pred: Tensor, img_meta: dict) → Tensor

Regress the bbox for the predicted class. Used in Cascade R-CNN.

Parameters

• priors (Tensor) – Priors from rpn_head or last stage bbox_head, has shape (num_proposals, 4).

• label (Tensor) – Only used when self.reg_class_agnostic is False, has shape (num_proposals, ).

• bbox_pred (Tensor) – Regression prediction of current stage bbox_head. When self.reg_class_agnostic is False, it has shape (n, num_classes * 4), otherwise it has shape (n, 4).

• img_meta (dict) – Image meta info.

Returns

Regressed bboxes, the same shape as input rois.

Return type

Tensor

class mmdet.models.roi_heads.BaseRoIExtractor(roi_layer: Union[ConfigDict, dict], out_channels: int, featmap_strides: List[int], init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Base class for RoI extractor.

Parameters

• roi_layer (ConfigDict or dict) – Specify RoI layer type and arguments.

• out_channels (int) – Output channels of RoI layers.

• featmap_strides (list[int]) – Strides of input feature maps.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict. Defaults to None.

build_roi_layers(layer_cfg: Union[ConfigDict, dict], featmap_strides: List[int]) → ModuleList

Build RoI operator to extract feature from each level feature map.

Parameters

• layer_cfg (ConfigDict or dict) – Dictionary to construct and config RoI layer operation. Options are modules under mmcv/ops such as RoIAlign.

• featmap_strides (list[int]) – The stride of input feature map w.r.t to the original image size, which would be used to scale RoI coordinate (original image coordinate system) to feature coordinate system.

Returns

The RoI extractor modules for each level

feature map.

Return type

nn.ModuleList

abstract forward(feats: Tuple[Tensor], rois: Tensor, roi_scale_factor: Optional[float] = None) → Tensor

Extractor ROI feats.

Parameters

• feats (Tuple[Tensor]) – Multi-scale features.

• rois (Tensor) – RoIs with the shape (n, 5) where the first column indicates batch id of each RoI.

• roi_scale_factor (Optional[float]) – RoI scale factor. Defaults to None.

Returns

RoI feature.

Return type

Tensor

property num_inputs: int

Number of input feature maps.

Type

int

roi_rescale(rois: Tensor, scale_factor: float) → Tensor

Scale RoI coordinates by scale factor.

Parameters

• rois (Tensor) – RoI (Region of Interest), shape (n, 5)

• scale_factor (float) – Scale factor that RoI will be multiplied by.

Returns

Scaled RoI.

Return type

Tensor

class mmdet.models.roi_heads.BaseRoIHead(bbox_roi_extractor: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, bbox_head: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, mask_roi_extractor: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, mask_head: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, shared_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Base class for RoIHeads.

abstract init_assigner_sampler(*args, **kwargs)

Initialize assigner and sampler.

abstract init_bbox_head(*args, **kwargs)

Initialize bbox_head

abstract init_mask_head(*args, **kwargs)

Initialize mask_head

abstract loss(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample])

Perform forward propagation and loss calculation of the roi head on the features of the upstream network.

predict(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the roi head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Features from upstream network. Each has shape (N, C, H, W).

• rpn_results_list (list[InstanceData]) – list of region proposals.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results to the original image. Defaults to True.

Returns

InstanceData]: Detection results of each image. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[obj

property with_bbox: bool

whether the RoI head contains a bbox_head

Type

bool

property with_mask: bool

whether the RoI head contains a mask_head

Type

bool

property with_shared_head: bool

whether the RoI head contains a shared_head

Type

bool

class mmdet.models.roi_heads.CascadeRoIHead(num_stages: int, stage_loss_weights: Union[List[float], Tuple[float]], bbox_roi_extractor: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, bbox_head: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, mask_roi_extractor: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, mask_head: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, shared_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Cascade roi head including one bbox head and one mask head.

https://arxiv.org/abs/1712.00726

bbox_loss(stage: int, x: Tuple[Tensor], sampling_results: List[SamplingResult]) → dict

Run forward function and calculate loss for box head in training.

Parameters

• stage (int) – The current stage in Cascade RoI Head.

• x (tuple[Tensor]) – List of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

Returns

Usually returns a dictionary with keys:

• cls_score (Tensor): Classification scores.

• bbox_pred (Tensor): Box energies / deltas.

• bbox_feats (Tensor): Extract bbox RoI features.

• loss_bbox (dict): A dictionary of bbox loss components.

• rois (Tensor): RoIs with the shape (n, 5) where the first column indicates batch id of each RoI.

• bbox_targets (tuple): Ground truth for proposals in a single image. Containing the following list of Tensors: (labels, label_weights, bbox_targets, bbox_weights)

Return type

dict

forward(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → tuple

Network forward process. Usually includes backbone, neck and head forward without any post-processing.

Parameters

• x (List[Tensor]) – Multi-level features that may have different resolutions.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – Each item contains the meta information of each image and corresponding annotations.

Returns

tuple: A tuple of features from bbox_head and mask_head forward.

init_assigner_sampler() → None

Initialize assigner and sampler for each stage.

init_bbox_head(bbox_roi_extractor: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]], bbox_head: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]) → None

Initialize box head and box roi extractor.

Parameters

• bbox_roi_extractor (ConfigDict, dict or list) – Config of box roi extractor.

• bbox_head (ConfigDict, dict or list) – Config of box in box head.

init_mask_head(mask_roi_extractor: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]], mask_head: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]) → None

Initialize mask head and mask roi extractor.

Parameters

• mask_head (dict) – Config of mask in mask head.

• mask_roi_extractor (ConfigDict, dict or list) – Config of mask roi extractor.

loss(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection roi on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict[str, Tensor]

mask_loss(stage: int, x: Tuple[Tensor], sampling_results: List[SamplingResult], batch_gt_instances: List[InstanceData]) → dict

Run forward function and calculate loss for mask head in training.

Parameters

• stage (int) – The current stage in Cascade RoI Head.

• x (tuple[Tensor]) – Tuple of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

Returns

Usually returns a dictionary with keys:

• mask_preds (Tensor): Mask prediction.

• loss_mask (dict): A dictionary of mask loss components.

Return type

dict

predict_bbox(x: Tuple[Tensor], batch_img_metas: List[dict], rpn_results_list: List[InstanceData], rcnn_test_cfg: Union[ConfigDict, dict], rescale: bool = False, **kwargs) → List[InstanceData]

Perform forward propagation of the bbox head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• (obj (rcnn_test_cfg) – ConfigDict): test_cfg of R-CNN.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

predict_mask(x: Tuple[Tensor], batch_img_metas: List[dict], results_list: List[InstanceData], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the mask head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• results_list (list[InstanceData]) – Detection results of each image.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[InstanceData]

class mmdet.models.roi_heads.CoarseMaskHead(num_convs: int = 0, num_fcs: int = 2, fc_out_channels: int = 1024, downsample_factor: int = 2, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'override': [{'name': 'fcs'}, {'type': 'Constant', 'val': 0.001, 'name': 'fc_logits'}], 'type': 'Xavier'}, *arg, **kwarg)

Coarse mask head used in PointRend.

Compared with standard FCNMaskHead, CoarseMaskHead will downsample the input feature map instead of upsample it.

Parameters

• num_convs (int) – Number of conv layers in the head. Defaults to 0.

• num_fcs (int) – Number of fc layers in the head. Defaults to 2.

• fc_out_channels (int) – Number of output channels of fc layer. Defaults to 1024.

• downsample_factor (int) – The factor that feature map is downsampled by. Defaults to 2.

• init_cfg (dict or list[dict], optional) – Initialization config dict.

forward(x: Tensor) → Tensor

Forward features from the upstream network.

Parameters

x (Tensor) – Extract mask RoI features.

Returns

Predicted foreground masks.

Return type

Tensor

init_weights() → None

Initialize weights.

class mmdet.models.roi_heads.ConvFCBBoxHead(num_shared_convs: int = 0, num_shared_fcs: int = 0, num_cls_convs: int = 0, num_cls_fcs: int = 0, num_reg_convs: int = 0, num_reg_fcs: int = 0, conv_out_channels: int = 256, fc_out_channels: int = 1024, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict]] = None, *args, **kwargs)

More general bbox head, with shared conv and fc layers and two optional separated branches.

                            /-> cls convs -> cls fcs -> cls
shared convs -> shared fcs
                            \-> reg convs -> reg fcs -> reg

forward(x: Tuple[Tensor]) → tuple

Forward features from the upstream network.

Parameters

x (tuple[Tensor]) – Features from the upstream network, each is a 4D-tensor.

Returns

A tuple of classification scores and bbox prediction.

• cls_score (Tensor): Classification scores for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * num_classes.

• bbox_pred (Tensor): Box energies / deltas for all scale levels, each is a 4D-tensor, the channels number is num_base_priors * 4.

Return type

tuple

class mmdet.models.roi_heads.DIIHead(num_classes: int = 80, num_ffn_fcs: int = 2, num_heads: int = 8, num_cls_fcs: int = 1, num_reg_fcs: int = 3, feedforward_channels: int = 2048, in_channels: int = 256, dropout: float = 0.0, ffn_act_cfg: Union[ConfigDict, dict] = {'inplace': True, 'type': 'ReLU'}, dynamic_conv_cfg: Union[ConfigDict, dict] = {'act_cfg': {'inplace': True, 'type': 'ReLU'}, 'feat_channels': 64, 'in_channels': 256, 'input_feat_shape': 7, 'norm_cfg': {'type': 'LN'}, 'out_channels': 256, 'type': 'DynamicConv'}, loss_iou: Union[ConfigDict, dict] = {'loss_weight': 2.0, 'type': 'GIoULoss'}, init_cfg: Optional[Union[ConfigDict, dict]] = None, **kwargs)

Dynamic Instance Interactive Head for Sparse R-CNN: End-to-End Object Detection with Learnable Proposals

Parameters

• num_classes (int) – Number of class in dataset. Defaults to 80.

• num_ffn_fcs (int) – The number of fully-connected layers in FFNs. Defaults to 2.

• num_heads (int) – The hidden dimension of FFNs. Defaults to 8.

• num_cls_fcs (int) – The number of fully-connected layers in classification subnet. Defaults to 1.

• num_reg_fcs (int) – The number of fully-connected layers in regression subnet. Defaults to 3.

• feedforward_channels (int) – The hidden dimension of FFNs. Defaults to 2048

• in_channels (int) – Hidden_channels of MultiheadAttention. Defaults to 256.

• dropout (float) – Probability of drop the channel. Defaults to 0.0

• ffn_act_cfg (ConfigDict or dict) – The activation config for FFNs.

• dynamic_conv_cfg (ConfigDict or dict) – The convolution config for DynamicConv.

• loss_iou (ConfigDict or dict) – The config for iou or giou loss.

:param init_cfg (ConfigDict or dict or list[ConfigDict or : dict]): Initialization config dict. Defaults to None.

forward(roi_feat: Tensor, proposal_feat: Tensor) → tuple

Forward function of Dynamic Instance Interactive Head.

Parameters

• roi_feat (Tensor) – Roi-pooling features with shape (batch_size*num_proposals, feature_dimensions, pooling_h , pooling_w).

• proposal_feat (Tensor) – Intermediate feature get from diihead in last stage, has shape (batch_size, num_proposals, feature_dimensions)

Returns

Usually a tuple of classification scores and bbox prediction and a intermediate feature.

• cls_scores (Tensor): Classification scores for all proposals, has shape (batch_size, num_proposals, num_classes).

• bbox_preds (Tensor): Box energies / deltas for all proposals, has shape (batch_size, num_proposals, 4).

• obj_feat (Tensor): Object feature before classification and regression subnet, has shape (batch_size, num_proposal, feature_dimensions).

• attn_feats (Tensor): Intermediate feature.

Return type

tuple[Tensor]

get_targets(sampling_results: List[SamplingResult], rcnn_train_cfg: ConfigDict, concat: bool = True) → tuple

Calculate the ground truth for all samples in a batch according to the sampling_results.

Almost the same as the implementation in bbox_head, we passed additional parameters pos_inds_list and neg_inds_list to _get_targets_single function.

Parameters

• (List[obj (sampling_results) – SamplingResult]): Assign results of all images in a batch after sampling.

• (obj (rcnn_train_cfg) – ConfigDict): train_cfg of RCNN.

• concat (bool) – Whether to concatenate the results of all the images in a single batch.

Returns

Ground truth for proposals in a single image. Containing the following list of Tensors:

• labels (list[Tensor],Tensor): Gt_labels for all proposals in a batch, each tensor in list has shape (num_proposals,) when concat=False, otherwise just a single tensor has shape (num_all_proposals,).

• label_weights (list[Tensor]): Labels_weights for all proposals in a batch, each tensor in list has shape (num_proposals,) when concat=False, otherwise just a single tensor has shape (num_all_proposals,).

• bbox_targets (list[Tensor],Tensor): Regression target for all proposals in a batch, each tensor in list has shape (num_proposals, 4) when concat=False, otherwise just a single tensor has shape (num_all_proposals, 4), the last dimension 4 represents [tl_x, tl_y, br_x, br_y].

• bbox_weights (list[tensor],Tensor): Regression weights for all proposals in a batch, each tensor in list has shape (num_proposals, 4) when concat=False, otherwise just a single tensor has shape (num_all_proposals, 4).

Return type

Tuple[Tensor]

init_weights() → None

Use xavier initialization for all weight parameter and set classification head bias as a specific value when use focal loss.

loss_and_target(cls_score: Tensor, bbox_pred: Tensor, sampling_results: List[SamplingResult], rcnn_train_cfg: Union[ConfigDict, dict], imgs_whwh: Tensor, concat: bool = True, reduction_override: Optional[str] = None) → dict

Calculate the loss based on the features extracted by the DIIHead.

Parameters

• cls_score (Tensor) – Classification prediction results of all class, has shape (batch_size * num_proposals_single_image, num_classes)

• bbox_pred (Tensor) – Regression prediction results, has shape (batch_size * num_proposals_single_image, 4), the last dimension 4 represents [tl_x, tl_y, br_x, br_y].

• (List[obj (sampling_results) – SamplingResult]): Assign results of all images in a batch after sampling.

• (obj (rcnn_train_cfg) – ConfigDict): train_cfg of RCNN.

• imgs_whwh (Tensor) – imgs_whwh (Tensor): Tensor with shape (batch_size, num_proposals, 4), the last dimension means [img_width,img_height, img_width, img_height].

• concat (bool) – Whether to concatenate the results of all the images in a single batch. Defaults to True.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”. Defaults to None.

Returns

A dictionary of loss and targets components. The targets are only used for cascade rcnn.

Return type

dict

class mmdet.models.roi_heads.DoubleConvFCBBoxHead(num_convs: int = 0, num_fcs: int = 0, conv_out_channels: int = 1024, fc_out_channels: int = 1024, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'type': 'BN'}, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'override': [{'type': 'Normal', 'name': 'fc_cls', 'std': 0.01}, {'type': 'Normal', 'name': 'fc_reg', 'std': 0.001}, {'type': 'Xavier', 'name': 'fc_branch', 'distribution': 'uniform'}], 'type': 'Normal'}, **kwargs)

Bbox head used in Double-Head R-CNN.

                                  /-> cls
              /-> shared convs ->
                                  \-> reg
roi features
                                  /-> cls
              \-> shared fc    ->
                                  \-> reg

forward(x_cls: Tensor, x_reg: Tensor) → Tuple[Tensor]

Forward features from the upstream network.

Parameters

• x_cls (Tensor) – Classification features of rois

• x_reg (Tensor) – Regression features from the upstream network.

Returns

A tuple of classification scores and bbox prediction.

• cls_score (Tensor): Classification score predictions of rois. each roi predicts num_classes + 1 channels.

• bbox_pred (Tensor): BBox deltas predictions of rois. each roi predicts 4 * num_classes channels.

Return type

tuple

class mmdet.models.roi_heads.DoubleHeadRoIHead(reg_roi_scale_factor: float, **kwargs)

RoI head for Double Head RCNN.

Parameters

reg_roi_scale_factor (float) – The scale factor to extend the rois used to extract the regression features.

class mmdet.models.roi_heads.DynamicRoIHead(**kwargs)

RoI head for Dynamic R-CNN.

bbox_loss(x: Tuple[Tensor], sampling_results: List[SamplingResult]) → dict

Perform forward propagation and loss calculation of the bbox head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

Returns

Usually returns a dictionary with keys:

• cls_score (Tensor): Classification scores.

• bbox_pred (Tensor): Box energies / deltas.

• bbox_feats (Tensor): Extract bbox RoI features.

• loss_bbox (dict): A dictionary of bbox loss components.

Return type

dict[str, Tensor]

loss(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → dict

Forward function for training.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

a dictionary of loss components

Return type

dict[str, Tensor]

update_hyperparameters()

Update hyperparameters like IoU thresholds for assigner and beta for SmoothL1 loss based on the training statistics.

Returns

the updated iou_thr and beta.

Return type

tuple[float]

class mmdet.models.roi_heads.FCNMaskHead(num_convs: int = 4, roi_feat_size: int = 14, in_channels: int = 256, conv_kernel_size: int = 3, conv_out_channels: int = 256, num_classes: int = 80, class_agnostic: int = False, upsample_cfg: Union[ConfigDict, dict] = {'scale_factor': 2, 'type': 'deconv'}, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, predictor_cfg: Union[ConfigDict, dict] = {'type': 'Conv'}, loss_mask: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_mask': True}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

forward(x: Tensor) → Tensor

Forward features from the upstream network.

Parameters

x (Tensor) – Extract mask RoI features.

Returns

Predicted foreground masks.

Return type

Tensor

get_targets(sampling_results: List[SamplingResult], batch_gt_instances: List[InstanceData], rcnn_train_cfg: ConfigDict) → Tensor

Calculate the ground truth for all samples in a batch according to the sampling_results.

Parameters

• (List[obj (sampling_results) – SamplingResult]): Assign results of all images in a batch after sampling.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

• (obj (rcnn_train_cfg) – ConfigDict): train_cfg of RCNN.

Returns

Mask target of each positive proposals in the image.

Return type

Tensor

init_weights() → None

Initialize the weights.

loss_and_target(mask_preds: Tensor, sampling_results: List[SamplingResult], batch_gt_instances: List[InstanceData], rcnn_train_cfg: ConfigDict) → dict

Calculate the loss based on the features extracted by the mask head.

Parameters

• mask_preds (Tensor) – Predicted foreground masks, has shape (num_pos, num_classes, h, w).

• (List[obj (sampling_results) – SamplingResult]): Assign results of all images in a batch after sampling.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

• (obj (rcnn_train_cfg) – ConfigDict): train_cfg of RCNN.

Returns

A dictionary of loss and targets components.

Return type

dict

predict_by_feat(mask_preds: Tuple[Tensor], results_list: List[InstanceData], batch_img_metas: List[dict], rcnn_test_cfg: ConfigDict, rescale: bool = False, activate_map: bool = False) → List[InstanceData]

Transform a batch of output features extracted from the head into mask results.

Parameters

• mask_preds (tuple[Tensor]) – Tuple of predicted foreground masks, each has shape (n, num_classes, h, w).

• results_list (list[InstanceData]) – Detection results of each image.

• batch_img_metas (list[dict]) – List of image information.

• (obj (rcnn_test_cfg) – ConfigDict): test_cfg of Bbox Head.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

• activate_map (book) – Whether get results with augmentations test. If True, the mask_preds will not process with sigmoid. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[InstanceData]

class mmdet.models.roi_heads.FeatureRelayHead(in_channels: int = 1024, out_conv_channels: int = 256, roi_feat_size: int = 7, scale_factor: int = 2, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'layer': 'Linear', 'type': 'Kaiming'})

Feature Relay Head used in SCNet.

Parameters

• in_channels (int) – number of input channels. Defaults to 256.

• conv_out_channels (int) – number of output channels before classification layer. Defaults to 256.

• roi_feat_size (int) – roi feat size at box head. Default: 7.

• scale_factor (int) – scale factor to match roi feat size at mask head. Defaults to 2.

:param init_cfg (ConfigDict or dict or list[dict] or: list[ConfigDict]): Initialization config dict. Defaults to

dict(type=’Kaiming’, layer=’Linear’).

forward(x: Tensor) → Optional[Tensor]

Forward function.

Parameters

x (Tensor) – Input feature.

Returns

Output feature. When the first dim of input is 0, None is returned.

Return type

Optional[Tensor]

class mmdet.models.roi_heads.FusedSemanticHead(num_ins: int, fusion_level: int, seg_scale_factor=0.125, num_convs: int = 4, in_channels: int = 256, conv_out_channels: int = 256, num_classes: int = 183, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, ignore_label: Optional[int] = None, loss_weight: Optional[float] = None, loss_seg: ConfigDict = {'ignore_index': 255, 'loss_weight': 0.2, 'type': 'CrossEntropyLoss'}, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'override': {'name': 'conv_logits'}, 'type': 'Kaiming'})

Multi-level fused semantic segmentation head.

in_1 -> 1x1 conv ---
                    |
in_2 -> 1x1 conv -- |
                   ||
in_3 -> 1x1 conv - ||
                  |||                  /-> 1x1 conv (mask prediction)
in_4 -> 1x1 conv -----> 3x3 convs (*4)
                    |                  \-> 1x1 conv (feature)
in_5 -> 1x1 conv ---

forward(feats: Tuple[Tensor]) → Tuple[Tensor]

Forward function.

Parameters

feats (tuple[Tensor]) – Multi scale feature maps.

Returns

• mask_preds (Tensor): Predicted mask logits.

• x (Tensor): Fused feature.

Return type

tuple[Tensor]

loss(mask_preds: Tensor, labels: Tensor) → Tensor

Loss function.

Parameters

• mask_preds (Tensor) – Predicted mask logits.

• labels (Tensor) – Ground truth.

Returns

Semantic segmentation loss.

Return type

Tensor

class mmdet.models.roi_heads.GenericRoIExtractor(aggregation: str = 'sum', pre_cfg: Optional[Union[ConfigDict, dict]] = None, post_cfg: Optional[Union[ConfigDict, dict]] = None, **kwargs)

Extract RoI features from all level feature maps levels.

This is the implementation of A novel Region of Interest Extraction Layer for Instance Segmentation.

Parameters

• aggregation (str) – The method to aggregate multiple feature maps. Options are ‘sum’, ‘concat’. Defaults to ‘sum’.

• pre_cfg (ConfigDict or dict) – Specify pre-processing modules. Defaults to None.

• post_cfg (ConfigDict or dict) – Specify post-processing modules. Defaults to None.

• kwargs (keyword arguments) – Arguments that are the same as BaseRoIExtractor.

forward(feats: Tuple[Tensor], rois: Tensor, roi_scale_factor: Optional[float] = None) → Tensor

Extractor ROI feats.

Parameters

• feats (Tuple[Tensor]) – Multi-scale features.

• rois (Tensor) – RoIs with the shape (n, 5) where the first column indicates batch id of each RoI.

• roi_scale_factor (Optional[float]) – RoI scale factor. Defaults to None.

Returns

RoI feature.

Return type

Tensor

class mmdet.models.roi_heads.GlobalContextHead(num_convs: int = 4, in_channels: int = 256, conv_out_channels: int = 256, num_classes: int = 80, loss_weight: float = 1.0, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, conv_to_res: bool = False, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'override': {'name': 'fc'}, 'std': 0.01, 'type': 'Normal'})

Global context head used in SCNet.

Parameters

• num_convs (int, optional) – number of convolutional layer in GlbCtxHead. Defaults to 4.

• in_channels (int, optional) – number of input channels. Defaults to 256.

• conv_out_channels (int, optional) – number of output channels before classification layer. Defaults to 256.

• num_classes (int, optional) – number of classes. Defaults to 80.

• loss_weight (float, optional) – global context loss weight. Defaults to 1.

• conv_cfg (dict, optional) – config to init conv layer. Defaults to None.

• norm_cfg (dict, optional) – config to init norm layer. Defaults to None.

• conv_to_res (bool, optional) – if True, 2 convs will be grouped into 1 SimplifiedBasicBlock using a skip connection. Defaults to False.

:param init_cfg (ConfigDict or dict or list[dict] or: list[ConfigDict]): Initialization config dict. Defaults to

dict(type=’Normal’, std=0.01, override=dict(name=’fc’)).

forward(feats: Tuple[Tensor]) → Tuple[Tensor]

Forward function.

Parameters

feats (Tuple[Tensor]) – Multi-scale feature maps.

Returns

• mc_pred (Tensor): Multi-class prediction.

• x (Tensor): Global context feature.

Return type

Tuple[Tensor]

loss(pred: Tensor, labels: List[Tensor]) → Tensor

Loss function.

Parameters

• pred (Tensor) – Logits.

• labels (list[Tensor]) – Grouth truths.

Returns

Loss.

Return type

Tensor

class mmdet.models.roi_heads.GridHead(grid_points: int = 9, num_convs: int = 8, roi_feat_size: int = 14, in_channels: int = 256, conv_kernel_size: int = 3, point_feat_channels: int = 64, deconv_kernel_size: int = 4, class_agnostic: bool = False, loss_grid: Union[ConfigDict, dict] = {'loss_weight': 15, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Union[ConfigDict, dict] = {'num_groups': 36, 'type': 'GN'}, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = [{'type': 'Kaiming', 'layer': ['Conv2d', 'Linear']}, {'type': 'Normal', 'layer': 'ConvTranspose2d', 'std': 0.001, 'override': {'type': 'Normal', 'name': 'deconv2', 'std': 0.001, 'bias': -4.59511985013459}}])

Implementation of Grid Head

Parameters

• grid_points (int) – The number of grid points. Defaults to 9.

• num_convs (int) – The number of convolution layers. Defaults to 8.

• roi_feat_size (int) – RoI feature size. Default to 14.

• in_channels (int) – The channel number of inputs features. Defaults to 256.

• conv_kernel_size (int) – The kernel size of convolution layers. Defaults to 3.

• point_feat_channels (int) – The number of channels of each point features. Defaults to 64.

• class_agnostic (bool) – Whether use class agnostic classification. If so, the output channels of logits will be 1. Defaults to False.

• loss_grid (ConfigDict or dict) – Config of grid loss.

• conv_cfg (ConfigDict or dict, optional) – construct and config conv layer.

• norm_cfg (ConfigDict or dict) – dictionary to construct and config norm layer.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict]) – Initialization config dict.

calc_sub_regions() → List[Tuple[float]]

Compute point specific representation regions.

See Grid R-CNN Plus for details.

forward(x: Tensor) → Dict[str, Tensor]

Forward function of GridHead.

Parameters

x (Tensor) – RoI features, has shape (num_rois, num_channels, roi_feat_size, roi_feat_size).

Returns

Return a dict including fused and unfused heatmap.

Return type

Dict[str, Tensor]

get_targets(sampling_results: List[SamplingResult], rcnn_train_cfg: ConfigDict) → Tensor

Calculate the ground truth for all samples in a batch according to the sampling_results.”.

Parameters

• sampling_results (List[SamplingResult]) – Assign results of all images in a batch after sampling.

• rcnn_train_cfg (ConfigDict) – train_cfg of RCNN.

Returns

Grid heatmap targets.

Return type

Tensor

loss(grid_pred: Tensor, sample_idx: Tensor, sampling_results: List[SamplingResult], rcnn_train_cfg: ConfigDict) → dict

Calculate the loss based on the features extracted by the grid head.

Parameters

• grid_pred (dict[str, Tensor]) – Outputs of grid_head forward.

• sample_idx (Tensor) – The sampling index of grid_pred.

• (List[obj (sampling_results) – SamplingResult]): Assign results of all images in a batch after sampling.

• (obj (rcnn_train_cfg) – ConfigDict): train_cfg of RCNN.

Returns

A dictionary of loss and targets components.

Return type

dict

predict_by_feat(grid_preds: Dict[str, Tensor], results_list: List[InstanceData], batch_img_metas: List[dict], rescale: bool = False) → List[InstanceData]

Adjust the predicted bboxes from bbox head.

Parameters

• grid_preds (dict[str, Tensor]) – dictionary outputted by forward function.

• results_list (list[InstanceData]) – Detection results of each image.

• batch_img_metas (list[dict]) – List of image information.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.roi_heads.GridRoIHead(grid_roi_extractor: Union[ConfigDict, dict], grid_head: Union[ConfigDict, dict], **kwargs)

Implementation of Grid RoI Head

Parameters

• grid_roi_extractor (ConfigDict or dict) – Config of roi extractor.

• grid_head (ConfigDict or dict) – Config of grid head

bbox_loss(x: Tuple[Tensor], sampling_results: List[SamplingResult], batch_img_metas: Optional[List[dict]] = None) → dict

Perform forward propagation and loss calculation of the bbox head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• sampling_results (list[SamplingResult]) – Sampling results.

• batch_img_metas (list[dict], optional) – Meta information of each image, e.g., image size, scaling factor, etc.

Returns

Usually returns a dictionary with keys:

• cls_score (Tensor): Classification scores.

• bbox_pred (Tensor): Box energies / deltas.

• bbox_feats (Tensor): Extract bbox RoI features.

• loss_bbox (dict): A dictionary of bbox loss components.

Return type

dict[str, Tensor]

forward(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: Optional[List[DetDataSample]] = None) → tuple

Network forward process. Usually includes backbone, neck and head forward without any post-processing.

Parameters

• x (Tuple[Tensor]) – Multi-level features that may have different resolutions.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – Each item contains

• corresponding (the meta information of each image and) –

• annotations. –

Returns

tuple: A tuple of features from bbox_head and mask_head forward.

loss(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample], **kwargs) → dict

Perform forward propagation and loss calculation of the detection roi on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict[str, Tensor]

predict_bbox(x: Tuple[Tensor], batch_img_metas: List[dict], rpn_results_list: List[InstanceData], rcnn_test_cfg: Union[ConfigDict, dict], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the bbox head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• rcnn_test_cfg (ConfigDict) – test_cfg of R-CNN.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.roi_heads.HTCMaskHead(with_conv_res: bool = True, *args, **kwargs)

Mask head for HTC.

Parameters

with_conv_res (bool) – Whether add conv layer for res_feat. Defaults to True.

forward(x: Tensor, res_feat: Optional[Tensor] = None, return_logits: bool = True, return_feat: bool = True) → Union[Tensor, List[Tensor]]

Parameters

• x (Tensor) – Feature map.

• res_feat (Tensor, optional) – Feature for residual connection. Defaults to None.

• return_logits (bool) – Whether return mask logits. Defaults to True.

• return_feat (bool) – Whether return feature map. Defaults to True.

Returns

The return result is one of three

results: res_feat, logits, or [logits, res_feat].

Return type

Union[Tensor, List[Tensor]]

class mmdet.models.roi_heads.HybridTaskCascadeRoIHead(num_stages: int, stage_loss_weights: List[float], semantic_roi_extractor: Optional[Union[ConfigDict, dict]] = None, semantic_head: Optional[Union[ConfigDict, dict]] = None, semantic_fusion: Tuple[str] = ('bbox', 'mask'), interleaved: bool = True, mask_info_flow: bool = True, **kwargs)

Hybrid task cascade roi head including one bbox head and one mask head.

https://arxiv.org/abs/1901.07518

Parameters

• num_stages (int) – Number of cascade stages.

• stage_loss_weights (list[float]) – Loss weight for every stage.

• semantic_roi_extractor (ConfigDict or dict, optional) – Config of semantic roi extractor. Defaults to None.

• Semantic_head (ConfigDict or dict, optional) – Config of semantic head. Defaults to None.

• interleaved (bool) – Whether to interleaves the box branch and mask branch. If True, the mask branch can take the refined bounding box predictions. Defaults to True.

• mask_info_flow (bool) – Whether to turn on the mask information flow, which means that feeding the mask features of the preceding stage to the current stage. Defaults to True.

bbox_loss(stage: int, x: Tuple[Tensor], sampling_results: List[SamplingResult], semantic_feat: Optional[Tensor] = None) → dict

Run forward function and calculate loss for box head in training.

Parameters

• stage (int) – The current stage in Cascade RoI Head.

• x (tuple[Tensor]) – List of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

• semantic_feat (Tensor, optional) – Semantic feature. Defaults to None.

Returns

Usually returns a dictionary with keys:

• cls_score (Tensor): Classification scores.

• bbox_pred (Tensor): Box energies / deltas.

• bbox_feats (Tensor): Extract bbox RoI features.

• loss_bbox (dict): A dictionary of bbox loss components.

• rois (Tensor): RoIs with the shape (n, 5) where the first column indicates batch id of each RoI.

• bbox_targets (tuple): Ground truth for proposals in a single image. Containing the following list of Tensors: (labels, label_weights, bbox_targets, bbox_weights)

Return type

dict

forward(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → tuple

Network forward process. Usually includes backbone, neck and head forward without any post-processing.

Parameters

• x (List[Tensor]) – Multi-level features that may have different resolutions.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – Each item contains the meta information of each image and corresponding annotations.

Returns

tuple: A tuple of features from bbox_head and mask_head forward.

loss(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection roi on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict[str, Tensor]

mask_loss(stage: int, x: Tuple[Tensor], sampling_results: List[SamplingResult], batch_gt_instances: List[InstanceData], semantic_feat: Optional[Tensor] = None) → dict

Run forward function and calculate loss for mask head in training.

Parameters

• stage (int) – The current stage in Cascade RoI Head.

• x (tuple[Tensor]) – Tuple of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

• semantic_feat (Tensor, optional) – Semantic feature. Defaults to None.

Returns

Usually returns a dictionary with keys:

• mask_preds (Tensor): Mask prediction.

• loss_mask (dict): A dictionary of mask loss components.

Return type

dict

predict(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the roi head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Features from upstream network. Each has shape (N, C, H, W).

• rpn_results_list (list[InstanceData]) – list of region proposals.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results to the original image. Defaults to False.

Returns

InstanceData]: Detection results of each image. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape

  (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape

  (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4),

  the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[obj

predict_mask(x: Tuple[Tensor], semantic_heat: Tensor, batch_img_metas: List[dict], results_list: List[InstanceData], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the mask head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• semantic_feat (Tensor) – Semantic feature.

• batch_img_metas (list[dict]) – List of image information.

• results_list (list[InstanceData]) – Detection results of each image.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[InstanceData]

property with_semantic: bool

whether the head has semantic head

Type

bool

class mmdet.models.roi_heads.MaskIoUHead(num_convs: int = 4, num_fcs: int = 2, roi_feat_size: int = 14, in_channels: int = 256, conv_out_channels: int = 256, fc_out_channels: int = 1024, num_classes: int = 80, loss_iou: Union[ConfigDict, dict] = {'loss_weight': 0.5, 'type': 'MSELoss'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = [{'type': 'Kaiming', 'override': {'name': 'convs'}}, {'type': 'Caffe2Xavier', 'override': {'name': 'fcs'}}, {'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_mask_iou'}}])

Mask IoU Head.

This head predicts the IoU of predicted masks and corresponding gt masks.

Parameters

• num_convs (int) – The number of convolution layers. Defaults to 4.

• num_fcs (int) – The number of fully connected layers. Defaults to 2.

• roi_feat_size (int) – RoI feature size. Default to 14.

• in_channels (int) – The channel number of inputs features. Defaults to 256.

• conv_out_channels (int) – The feature channels of convolution layers. Defaults to 256.

• fc_out_channels (int) – The feature channels of fully connected layers. Defaults to 1024.

• num_classes (int) – Number of categories excluding the background category. Defaults to 80.

• loss_iou (ConfigDict or dict) – IoU loss.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict.

forward(mask_feat: Tensor, mask_preds: Tensor) → Tensor

Forward function.

Parameters

• mask_feat (Tensor) – Mask features from upstream models.

• mask_preds (Tensor) – Mask predictions from mask head.

Returns

Mask IoU predictions.

Return type

Tensor

get_targets(sampling_results: List[SamplingResult], batch_gt_instances: List[InstanceData], mask_preds: Tensor, mask_targets: Tensor, rcnn_train_cfg: ConfigDict) → Tensor

Compute target of mask IoU.

Mask IoU target is the IoU of the predicted mask (inside a bbox) and the gt mask of corresponding gt mask (the whole instance). The intersection area is computed inside the bbox, and the gt mask area is computed with two steps, firstly we compute the gt area inside the bbox, then divide it by the area ratio of gt area inside the bbox and the gt area of the whole instance.

Parameters

• sampling_results (list[SamplingResult]) – sampling results.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It includes masks inside.

• mask_preds (Tensor) – Predicted masks of each positive proposal, shape (num_pos, h, w).

• mask_targets (Tensor) – Gt mask of each positive proposal, binary map of the shape (num_pos, h, w).

• (obj (rcnn_train_cfg) – ConfigDict): Training config for R-CNN part.

Returns

mask iou target (length == num positive).

Return type

Tensor

loss_and_target(mask_iou_pred: Tensor, mask_preds: Tensor, mask_targets: Tensor, sampling_results: List[SamplingResult], batch_gt_instances: List[InstanceData], rcnn_train_cfg: ConfigDict) → dict

Calculate the loss and targets of MaskIoUHead.

Parameters

• mask_iou_pred (Tensor) – Mask IoU predictions results, has shape (num_pos, num_classes)

• mask_preds (Tensor) – Mask predictions from mask head, has shape (num_pos, mask_size, mask_size).

• mask_targets (Tensor) – The ground truth masks assigned with predictions, has shape (num_pos, mask_size, mask_size).

• (List[obj (sampling_results) – SamplingResult]): Assign results of all images in a batch after sampling.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It includes masks inside.

• (obj (rcnn_train_cfg) – ConfigDict): train_cfg of RCNN.

Returns

A dictionary of loss and targets components.

The targets are only used for cascade rcnn.

Return type

dict

predict_by_feat(mask_iou_preds: Tuple[Tensor], results_list: List[InstanceData]) → List[InstanceData]

Predict the mask iou and calculate it into results.scores.

Parameters

• mask_iou_preds (Tensor) – Mask IoU predictions results, has shape (num_proposals, num_classes)

• results_list (list[InstanceData]) – Detection results of each image.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[InstanceData]

class mmdet.models.roi_heads.MaskPointHead(num_classes: int, num_fcs: int = 3, in_channels: int = 256, fc_channels: int = 256, class_agnostic: bool = False, coarse_pred_each_layer: bool = True, conv_cfg: Union[ConfigDict, dict] = {'type': 'Conv1d'}, norm_cfg: Optional[Union[ConfigDict, dict]] = None, act_cfg: Union[ConfigDict, dict] = {'type': 'ReLU'}, loss_point: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_mask': True}, init_cfg: Union[ConfigDict, dict, List[Union[ConfigDict, dict]]] = {'override': {'name': 'fc_logits'}, 'std': 0.001, 'type': 'Normal'})

A mask point head use in PointRend.

MaskPointHead use shared multi-layer perceptron (equivalent to nn.Conv1d) to predict the logit of input points. The fine-grained feature and coarse feature will be concatenate together for predication.

Parameters

• num_fcs (int) – Number of fc layers in the head. Defaults to 3.

• in_channels (int) – Number of input channels. Defaults to 256.

• fc_channels (int) – Number of fc channels. Defaults to 256.

• num_classes (int) – Number of classes for logits. Defaults to 80.

• class_agnostic (bool) – Whether use class agnostic classification. If so, the output channels of logits will be 1. Defaults to False.

• coarse_pred_each_layer (bool) – Whether concatenate coarse feature with the output of each fc layer. Defaults to True.

• conv_cfg (ConfigDict or dict) – Dictionary to construct and config conv layer. Defaults to dict(type=’Conv1d’)).

• norm_cfg (ConfigDict or dict, optional) – Dictionary to construct and config norm layer. Defaults to None.

• loss_point (ConfigDict or dict) – Dictionary to construct and config loss layer of point head. Defaults to dict(type=’CrossEntropyLoss’, use_mask=True, loss_weight=1.0).

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict.

forward(fine_grained_feats: Tensor, coarse_feats: Tensor) → Tensor

Classify each point base on fine grained and coarse feats.

Parameters

• fine_grained_feats (Tensor) – Fine grained feature sampled from FPN, shape (num_rois, in_channels, num_points).

• coarse_feats (Tensor) – Coarse feature sampled from CoarseMaskHead, shape (num_rois, num_classes, num_points).

Returns

Point classification results, shape (num_rois, num_class, num_points).

Return type

Tensor

get_roi_rel_points_test(mask_preds: Tensor, label_preds: Tensor, cfg: Union[ConfigDict, dict]) → Tuple[Tensor, Tensor]

Get num_points most uncertain points during test.

Parameters

• mask_preds (Tensor) – A tensor of shape (num_rois, num_classes, mask_height, mask_width) for class-specific or class-agnostic prediction.

• label_preds (Tensor) – The predication class for each instance.

• cfg (ConfigDict or dict) – Testing config of point head.

Returns

• point_indices (Tensor): A tensor of shape (num_rois, num_points) that contains indices from [0, mask_height x mask_width) of the most uncertain points.

• point_coords (Tensor): A tensor of shape (num_rois, num_points, 2) that contains [0, 1] x [0, 1] normalized coordinates of the most uncertain points from the [mask_height, mask_width] grid.

Return type

tuple

get_roi_rel_points_train(mask_preds: Tensor, labels: Tensor, cfg: Union[ConfigDict, dict]) → Tensor

Get num_points most uncertain points with random points during train.

Sample points in [0, 1] x [0, 1] coordinate space based on their uncertainty. The uncertainties are calculated for each point using ‘_get_uncertainty()’ function that takes point’s logit prediction as input.

Parameters

• mask_preds (Tensor) – A tensor of shape (num_rois, num_classes, mask_height, mask_width) for class-specific or class-agnostic prediction.

• labels (Tensor) – The ground truth class for each instance.

• cfg (ConfigDict or dict) – Training config of point head.

Returns

A tensor of shape (num_rois, num_points, 2) that contains the coordinates sampled points.

Return type

point_coords (Tensor)

get_targets(rois: Tensor, rel_roi_points: Tensor, sampling_results: List[SamplingResult], batch_gt_instances: List[InstanceData], cfg: Union[ConfigDict, dict]) → Tensor

Get training targets of MaskPointHead for all images.

Parameters

• rois (Tensor) – Region of Interest, shape (num_rois, 5).

• rel_roi_points (Tensor) – Points coordinates relative to RoI, shape (num_rois, num_points, 2).

• sampling_results (SamplingResult) – Sampling result after sampling and assignment.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

• (obj (cfg) – ConfigDict or dict): Training cfg.

Returns

Point target, shape (num_rois, num_points).

Return type

Tensor

loss_and_target(point_pred: Tensor, rel_roi_points: Tensor, sampling_results: List[SamplingResult], batch_gt_instances: List[InstanceData], cfg: Union[ConfigDict, dict]) → dict

Calculate loss for MaskPointHead.

Parameters

• point_pred (Tensor) – Point predication result, shape (num_rois, num_classes, num_points).

• rel_roi_points (Tensor) –

  Points coordinates relative to RoI, shape

  (num_rois, num_points, 2).

  sampling_results (SamplingResult): Sampling result after

  sampling and assignment.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

• (obj (cfg) – ConfigDict or dict): Training cfg.

Returns

a dictionary of point loss and point target.

Return type

dict

class mmdet.models.roi_heads.MaskScoringRoIHead(mask_iou_head: Union[ConfigDict, dict], **kwargs)

Mask Scoring RoIHead for `Mask Scoring RCNN.

<https://arxiv.org/abs/1903.00241>`_.

Parameters

( (mask_iou_head) – obj`ConfigDict`, dict): The config of mask_iou_head.

forward(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: Optional[List[DetDataSample]] = None) → tuple

Network forward process. Usually includes backbone, neck and head forward without any post-processing.

Parameters

• x (List[Tensor]) – Multi-level features that may have different resolutions.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – Each item contains

• corresponding (the meta information of each image and) –

• annotations. –

Returns

tuple: A tuple of features from bbox_head and mask_head forward.

mask_loss(x: Tuple[Tensor], sampling_results: List[SamplingResult], bbox_feats, batch_gt_instances: List[InstanceData]) → dict

Perform forward propagation and loss calculation of the mask head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Tuple of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

• bbox_feats (Tensor) – Extract bbox RoI features.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

Returns

Usually returns a dictionary with keys:

• mask_preds (Tensor): Mask prediction.

• mask_feats (Tensor): Extract mask RoI features.

• mask_targets (Tensor): Mask target of each positive proposals in the image.

• loss_mask (dict): A dictionary of mask loss components.

• loss_mask_iou (Tensor): mask iou loss.

Return type

dict

predict_mask(x: Tensor, batch_img_metas: List[dict], results_list: List[InstanceData], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the mask head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• results_list (list[InstanceData]) – Detection results of each image.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[InstanceData]

class mmdet.models.roi_heads.MultiInstanceRoIHead(num_instance: int = 2, *args, **kwargs)

The roi head for Multi-instance prediction.

bbox_loss(x: Tuple[Tensor], sampling_results: List[SamplingResult]) → dict

Perform forward propagation and loss calculation of the bbox head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

Returns

Usually returns a dictionary with keys:

• cls_score (Tensor): Classification scores.

• bbox_pred (Tensor): Box energies / deltas.

• bbox_feats (Tensor): Extract bbox RoI features.

• loss_bbox (dict): A dictionary of bbox loss components.

Return type

dict[str, Tensor]

init_bbox_head(bbox_roi_extractor: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict]) → None

Initialize box head and box roi extractor.

Parameters

• bbox_roi_extractor (dict or ConfigDict) – Config of box roi extractor.

• bbox_head (dict or ConfigDict) – Config of box in box head.

loss(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection roi on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict[str, Tensor]

predict_bbox(x: Tuple[Tensor], batch_img_metas: List[dict], rpn_results_list: List[InstanceData], rcnn_test_cfg: Union[ConfigDict, dict], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the bbox head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• (obj (rcnn_test_cfg) – ConfigDict): test_cfg of R-CNN.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

class mmdet.models.roi_heads.PISARoIHead(bbox_roi_extractor: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, bbox_head: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, mask_roi_extractor: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, mask_head: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, shared_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

The RoI head for Prime Sample Attention in Object Detection.

bbox_loss(x: Tuple[Tensor], sampling_results: List[SamplingResult], neg_label_weights: Optional[List[Tensor]] = None) → dict

Perform forward propagation and loss calculation of the bbox head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

Returns

Usually returns a dictionary with keys:

• cls_score (Tensor): Classification scores.

• bbox_pred (Tensor): Box energies / deltas.

• bbox_feats (Tensor): Extract bbox RoI features.

• loss_bbox (dict): A dictionary of bbox loss components.

Return type

dict[str, Tensor]

loss(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection roi on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict[str, Tensor]

class mmdet.models.roi_heads.PointRendRoIHead(point_head: Union[ConfigDict, dict], *args, **kwargs)

PointRend.

init_point_head(point_head: Union[ConfigDict, dict]) → None

Initialize point_head

mask_loss(x: Tuple[Tensor], sampling_results: List[SamplingResult], bbox_feats: Tensor, batch_gt_instances: List[InstanceData]) → dict

Run forward function and calculate loss for mask head and point head in training.

predict_mask(x: Tuple[Tensor], batch_img_metas: List[dict], results_list: List[InstanceData], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the mask head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• results_list (list[InstanceData]) – Detection results of each image.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[InstanceData]

class mmdet.models.roi_heads.ResLayer(depth, stage=3, stride=2, dilation=1, style='pytorch', norm_cfg={'requires_grad': True, 'type': 'BN'}, norm_eval=True, with_cp=False, dcn=None, pretrained=None, init_cfg=None)

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

train(mode=True)

Set the module in training mode.

This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Parameters

mode (bool) – whether to set training mode (True) or evaluation mode (False). Default: True.

Returns

self

Return type

Module

class mmdet.models.roi_heads.SABLHead(num_classes: int, cls_in_channels: int = 256, reg_in_channels: int = 256, roi_feat_size: int = 7, reg_feat_up_ratio: int = 2, reg_pre_kernel: int = 3, reg_post_kernel: int = 3, reg_pre_num: int = 2, reg_post_num: int = 1, cls_out_channels: int = 1024, reg_offset_out_channels: int = 256, reg_cls_out_channels: int = 256, num_cls_fcs: int = 1, num_reg_fcs: int = 0, reg_class_agnostic: bool = True, norm_cfg: Optional[Union[ConfigDict, dict]] = None, bbox_coder: Union[ConfigDict, dict] = {'num_buckets': 14, 'scale_factor': 1.7, 'type': 'BucketingBBoxCoder'}, loss_cls: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': False}, loss_bbox_cls: Union[ConfigDict, dict] = {'loss_weight': 1.0, 'type': 'CrossEntropyLoss', 'use_sigmoid': True}, loss_bbox_reg: Union[ConfigDict, dict] = {'beta': 0.1, 'loss_weight': 1.0, 'type': 'SmoothL1Loss'}, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Side-Aware Boundary Localization (SABL) for RoI-Head.

Side-Aware features are extracted by conv layers with an attention mechanism. Boundary Localization with Bucketing and Bucketing Guided Rescoring are implemented in BucketingBBoxCoder.

Please refer to https://arxiv.org/abs/1912.04260 for more details.

Parameters

• cls_in_channels (int) – Input channels of cls RoI feature. Defaults to 256.

• reg_in_channels (int) – Input channels of reg RoI feature. Defaults to 256.

• roi_feat_size (int) – Size of RoI features. Defaults to 7.

• reg_feat_up_ratio (int) – Upsample ratio of reg features. Defaults to 2.

• reg_pre_kernel (int) – Kernel of 2D conv layers before attention pooling. Defaults to 3.

• reg_post_kernel (int) – Kernel of 1D conv layers after attention pooling. Defaults to 3.

• reg_pre_num (int) – Number of pre convs. Defaults to 2.

• reg_post_num (int) – Number of post convs. Defaults to 1.

• num_classes (int) – Number of classes in dataset. Defaults to 80.

• cls_out_channels (int) – Hidden channels in cls fcs. Defaults to 1024.

• reg_offset_out_channels (int) – Hidden and output channel of reg offset branch. Defaults to 256.

• reg_cls_out_channels (int) – Hidden and output channel of reg cls branch. Defaults to 256.

• num_cls_fcs (int) – Number of fcs for cls branch. Defaults to 1.

• num_reg_fcs (int) – Number of fcs for reg branch.. Defaults to 0.

• reg_class_agnostic (bool) – Class agnostic regression or not. Defaults to True.

• norm_cfg (dict) – Config of norm layers. Defaults to None.

• bbox_coder (dict) – Config of bbox coder. Defaults ‘BucketingBBoxCoder’.

• loss_cls (dict) – Config of classification loss.

• loss_bbox_cls (dict) – Config of classification loss for bbox branch.

• loss_bbox_reg (dict) – Config of regression loss for bbox branch.

• init_cfg (dict or list[dict], optional) – Initialization config dict. Defaults to None.

attention_pool(reg_x: Tensor) → tuple

Extract direction-specific features fx and fy with attention methanism.

bbox_pred_split(bbox_pred: tuple, num_proposals_per_img: Sequence[int]) → tuple

Split batch bbox prediction back to each image.

bucket_target(pos_proposals_list: list, neg_proposals_list: list, pos_gt_bboxes_list: list, pos_gt_labels_list: list, rcnn_train_cfg: ConfigDict, concat: bool = True) → tuple

Compute bucketing estimation targets and fine regression targets for a batch of images.

cls_forward(cls_x: Tensor) → Tensor

Forward of classification fc layers.

forward(x: Tensor) → tuple

Forward features from the upstream network.

get_targets(sampling_results: List[SamplingResult], rcnn_train_cfg: ConfigDict, concat: bool = True) → tuple

Calculate the ground truth for all samples in a batch according to the sampling_results.

loss(cls_score: Tensor, bbox_pred: Tuple[Tensor, Tensor], rois: Tensor, labels: Tensor, label_weights: Tensor, bbox_targets: Tuple[Tensor, Tensor], bbox_weights: Tuple[Tensor, Tensor], reduction_override: Optional[str] = None) → dict

Calculate the loss based on the network predictions and targets.

Parameters

• cls_score (Tensor) – Classification prediction results of all class, has shape (batch_size * num_proposals_single_image, num_classes)

• bbox_pred (Tensor) – A tuple of regression prediction results containing bucket_cls_preds and bucket_offset_preds.

• rois (Tensor) – RoIs with the shape (batch_size * num_proposals_single_image, 5) where the first column indicates batch id of each RoI.

• labels (Tensor) – Gt_labels for all proposals in a batch, has shape (batch_size * num_proposals_single_image, ).

• label_weights (Tensor) – Labels_weights for all proposals in a batch, has shape (batch_size * num_proposals_single_image, ).

• bbox_targets (Tuple[Tensor, Tensor]) – A tuple of regression target containing bucket_cls_targets and bucket_offset_targets. the last dimension 4 represents [tl_x, tl_y, br_x, br_y].

• bbox_weights (Tuple[Tensor, Tensor]) – A tuple of regression weights containing bucket_cls_weights and bucket_offset_weights.

• reduction_override (str, optional) – The reduction method used to override the original reduction method of the loss. Options are “none”, “mean” and “sum”. Defaults to None,

Returns

A dictionary of loss.

Return type

dict

refine_bboxes(sampling_results: List[SamplingResult], bbox_results: dict, batch_img_metas: List[dict]) → List[InstanceData]

Refine bboxes during training.

Parameters

• sampling_results (List[SamplingResult]) – Sampling results.

• bbox_results (dict) –

  Usually is a dictionary with keys:

  • cls_score (Tensor): Classification scores.

  • bbox_pred (Tensor): Box energies / deltas.

  • rois (Tensor): RoIs with the shape (n, 5) where the first column indicates batch id of each RoI.

  • bbox_targets (tuple): Ground truth for proposals in a single image. Containing the following list of Tensors: (labels, label_weights, bbox_targets, bbox_weights)

• batch_img_metas (List[dict]) – List of image information.

Returns

Refined bboxes of each image.

Return type

list[InstanceData]

reg_forward(reg_x: Tensor) → tuple

Forward of regression branch.

reg_pred(x: Tensor, offset_fcs: ModuleList, cls_fcs: ModuleList) → tuple

Predict bucketing estimation (cls_pred) and fine regression (offset pred) with side-aware features.

regress_by_class(rois: Tensor, label: Tensor, bbox_pred: tuple, img_meta: dict) → Tensor

Regress the bbox for the predicted class. Used in Cascade R-CNN.

Parameters

• rois (Tensor) – shape (n, 4) or (n, 5)

• label (Tensor) – shape (n, )

• bbox_pred (Tuple[Tensor]) – shape [(n, num_buckets *2), (n, num_buckets *2)]

• img_meta (dict) – Image meta info.

Returns

Regressed bboxes, the same shape as input rois.

Return type

Tensor

side_aware_feature_extractor(reg_x: Tensor) → tuple

Refine and extract side-aware features without split them.

side_aware_split(feat: Tensor) → Tensor

Split side-aware features aligned with orders of bucketing targets.

class mmdet.models.roi_heads.SCNetBBoxHead(num_shared_convs: int = 0, num_shared_fcs: int = 0, num_cls_convs: int = 0, num_cls_fcs: int = 0, num_reg_convs: int = 0, num_reg_fcs: int = 0, conv_out_channels: int = 256, fc_out_channels: int = 1024, conv_cfg: Optional[Union[ConfigDict, dict]] = None, norm_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict]] = None, *args, **kwargs)

BBox head for SCNet.

This inherits ConvFCBBoxHead with modified forward() function, allow us to get intermediate shared feature.

forward(x: Tensor, return_shared_feat: bool = False) → Union[Tensor, Tuple[Tensor]]

Forward function.

Parameters

• x (Tensor) – input features

• return_shared_feat (bool) – If True, return cls-reg-shared feature.

Returns

contain cls_score and bbox_pred,

if return_shared_feat is True, append x_shared to the returned tuple.

Return type

out (tuple[Tensor])

class mmdet.models.roi_heads.SCNetMaskHead(conv_to_res: bool = True, **kwargs)

Mask head for SCNet.

Parameters

conv_to_res (bool, optional) – if True, change the conv layers to SimplifiedBasicBlock.

class mmdet.models.roi_heads.SCNetRoIHead(num_stages: int, stage_loss_weights: List[float], semantic_roi_extractor: Optional[Union[ConfigDict, dict]] = None, semantic_head: Optional[Union[ConfigDict, dict]] = None, feat_relay_head: Optional[Union[ConfigDict, dict]] = None, glbctx_head: Optional[Union[ConfigDict, dict]] = None, **kwargs)

RoIHead for SCNet.

Parameters

• num_stages (int) – number of cascade stages.

• stage_loss_weights (list) – loss weight of cascade stages.

• semantic_roi_extractor (dict) – config to init semantic roi extractor.

• semantic_head (dict) – config to init semantic head.

• feat_relay_head (dict) – config to init feature_relay_head.

• glbctx_head (dict) – config to init global context head.

bbox_loss(stage: int, x: Tuple[Tensor], sampling_results: List[SamplingResult], semantic_feat: Optional[Tensor] = None, glbctx_feat: Optional[Tensor] = None) → dict

Run forward function and calculate loss for box head in training.

Parameters

• stage (int) – The current stage in Cascade RoI Head.

• x (tuple[Tensor]) – List of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

• semantic_feat (Tensor) – Semantic feature. Defaults to None.

• glbctx_feat (Tensor) – Global context feature. Defaults to None.

Returns

Usually returns a dictionary with keys:

• cls_score (Tensor): Classification scores.

• bbox_pred (Tensor): Box energies / deltas.

• bbox_feats (Tensor): Extract bbox RoI features.

• loss_bbox (dict): A dictionary of bbox loss components.

• rois (Tensor): RoIs with the shape (n, 5) where the first column indicates batch id of each RoI.

• bbox_targets (tuple): Ground truth for proposals in a single image. Containing the following list of Tensors: (labels, label_weights, bbox_targets, bbox_weights)

Return type

dict

forward(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → tuple

Network forward process. Usually includes backbone, neck and head forward without any post-processing.

Parameters

• x (List[Tensor]) – Multi-level features that may have different resolutions.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – Each item contains the meta information of each image and corresponding annotations.

Returns

tuple: A tuple of features from bbox_head and mask_head forward.

global_context_loss(x: Tuple[Tensor], batch_gt_instances: List[InstanceData]) → dict

Global context loss.

Parameters

• x (Tuple[Tensor]) – Tuple of multi-level img features.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

Returns

Usually returns a dictionary with keys:

• glbctx_feat (Tensor): Global context feature.

• loss_glbctx (dict): Global context loss.

Return type

dict

init_mask_head(mask_roi_extractor: Union[ConfigDict, dict], mask_head: Union[ConfigDict, dict]) → None

Initialize mask_head

loss(x: Tensor, rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection roi on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict[str, Tensor]

mask_loss(x: Tuple[Tensor], sampling_results: List[SamplingResult], batch_gt_instances: List[InstanceData], semantic_feat: Optional[Tensor] = None, glbctx_feat: Optional[Tensor] = None, relayed_feat: Optional[Tensor] = None) → dict

Run forward function and calculate loss for mask head in training.

Parameters

• x (tuple[Tensor]) – Tuple of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

• semantic_feat (Tensor) – Semantic feature. Defaults to None.

• glbctx_feat (Tensor) – Global context feature. Defaults to None.

• relayed_feat (Tensor) – Relayed feature. Defaults to None.

Returns

Usually returns a dictionary with keys:

• mask_preds (Tensor): Mask prediction.

• loss_mask (dict): A dictionary of mask loss components.

Return type

dict

predict(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the roi head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Features from upstream network. Each has shape (N, C, H, W).

• rpn_results_list (list[InstanceData]) – list of region proposals.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results to the original image. Defaults to False.

Returns

InstanceData]: Detection results of each image. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape

  (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape

  (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4),

  the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[obj

predict_mask(x: Tuple[Tensor], semantic_heat: Tensor, glbctx_feat: Tensor, batch_img_metas: List[dict], results_list: List[InstanceData], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the mask head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• semantic_feat (Tensor) – Semantic feature.

• glbctx_feat (Tensor) – Global context feature.

• batch_img_metas (list[dict]) – List of image information.

• results_list (list[InstanceData]) – Detection results of each image.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[InstanceData]

semantic_loss(x: Tuple[Tensor], batch_data_samples: List[DetDataSample]) → dict

Semantic segmentation loss.

Parameters

• x (Tuple[Tensor]) – Tuple of multi-level img features.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

Usually returns a dictionary with keys:

• semantic_feat (Tensor): Semantic feature.

• loss_seg (dict): Semantic segmentation loss.

Return type

dict

property with_feat_relay: bool

whether the head has feature relay head

Type

bool

property with_glbctx: bool

whether the head has global context head

Type

bool

property with_semantic: bool

whether the head has semantic head

Type

bool

class mmdet.models.roi_heads.SCNetSemanticHead(conv_to_res: bool = True, **kwargs)

Mask head for SCNet.

Parameters

conv_to_res (bool, optional) – if True, change the conv layers to SimplifiedBasicBlock.

class mmdet.models.roi_heads.Shared2FCBBoxHead(fc_out_channels: int = 1024, *args, **kwargs)

class mmdet.models.roi_heads.Shared4Conv1FCBBoxHead(fc_out_channels: int = 1024, *args, **kwargs)

class mmdet.models.roi_heads.SingleRoIExtractor(roi_layer: Union[ConfigDict, dict], out_channels: int, featmap_strides: List[int], finest_scale: int = 56, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Extract RoI features from a single level feature map.

If there are multiple input feature levels, each RoI is mapped to a level according to its scale. The mapping rule is proposed in FPN.

Parameters

• roi_layer (ConfigDict or dict) – Specify RoI layer type and arguments.

• out_channels (int) – Output channels of RoI layers.

• featmap_strides (List[int]) – Strides of input feature maps.

• finest_scale (int) – Scale threshold of mapping to level 0. Defaults to 56.

• init_cfg (ConfigDict or dict or list[ConfigDict or dict], optional) – Initialization config dict. Defaults to None.

forward(feats: Tuple[Tensor], rois: Tensor, roi_scale_factor: Optional[float] = None)

Extractor ROI feats.

Parameters

• feats (Tuple[Tensor]) – Multi-scale features.

• rois (Tensor) – RoIs with the shape (n, 5) where the first column indicates batch id of each RoI.

• roi_scale_factor (Optional[float]) – RoI scale factor. Defaults to None.

Returns

RoI feature.

Return type

Tensor

map_roi_levels(rois: Tensor, num_levels: int) → Tensor

Map rois to corresponding feature levels by scales.

• scale < finest_scale * 2: level 0

• finest_scale * 2 <= scale < finest_scale * 4: level 1

• finest_scale * 4 <= scale < finest_scale * 8: level 2

• scale >= finest_scale * 8: level 3

Parameters

• rois (Tensor) – Input RoIs, shape (k, 5).

• num_levels (int) – Total level number.

Returns

Level index (0-based) of each RoI, shape (k, )

Return type

Tensor

class mmdet.models.roi_heads.SparseRoIHead(num_stages: int = 6, stage_loss_weights: Tuple[float] = (1, 1, 1, 1, 1, 1), proposal_feature_channel: int = 256, bbox_roi_extractor: Union[ConfigDict, dict] = {'featmap_strides': [4, 8, 16, 32], 'out_channels': 256, 'roi_layer': {'output_size': 7, 'sampling_ratio': 2, 'type': 'RoIAlign'}, 'type': 'SingleRoIExtractor'}, mask_roi_extractor: Optional[Union[ConfigDict, dict]] = None, bbox_head: Union[ConfigDict, dict] = {'dropout': 0.0, 'feedforward_channels': 2048, 'ffn_act_cfg': {'inplace': True, 'type': 'ReLU'}, 'hidden_channels': 256, 'num_classes': 80, 'num_cls_fcs': 1, 'num_fcs': 2, 'num_heads': 8, 'num_reg_fcs': 3, 'roi_feat_size': 7, 'type': 'DIIHead'}, mask_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict]] = None)

The RoIHead for Sparse R-CNN: End-to-End Object Detection with Learnable Proposals and Instances as Queries

Parameters

• num_stages (int) – Number of stage whole iterative process. Defaults to 6.

• stage_loss_weights (Tuple[float]) – The loss weight of each stage. By default all stages have the same weight 1.

• bbox_roi_extractor (ConfigDict or dict) – Config of box roi extractor.

• mask_roi_extractor (ConfigDict or dict) – Config of mask roi extractor.

• bbox_head (ConfigDict or dict) – Config of box head.

• mask_head (ConfigDict or dict) – Config of mask head.

• train_cfg (ConfigDict or dict, Optional) – Configuration information in train stage. Defaults to None.

• test_cfg (ConfigDict or dict, Optional) – Configuration information in test stage. Defaults to None.

:param init_cfg (ConfigDict or dict or list[ConfigDict or : dict]): Initialization config dict. Defaults to None.

bbox_loss(stage: int, x: Tuple[Tensor], results_list: List[InstanceData], object_feats: Tensor, batch_img_metas: List[dict], batch_gt_instances: List[InstanceData]) → dict

Perform forward propagation and loss calculation of the bbox head on the features of the upstream network.

Parameters

• stage (int) – The current stage in iterative process.

• x (tuple[Tensor]) – List of multi-level img features.

• results_list (List[InstanceData]) – List of region proposals.

• object_feats (Tensor) – The object feature extracted from the previous stage.

• batch_img_metas (list[dict]) – Meta information of each image.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

Returns

Usually returns a dictionary with keys:

• cls_score (Tensor): Classification scores.

• bbox_pred (Tensor): Box energies / deltas.

• bbox_feats (Tensor): Extract bbox RoI features.

• loss_bbox (dict): A dictionary of bbox loss components.

Return type

dict[str, Tensor]

forward(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → tuple

Network forward process. Usually includes backbone, neck and head forward without any post-processing.

Parameters

• x (List[Tensor]) – Multi-level features that may have different resolutions.

• rpn_results_list (List[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

tuple: A tuple of features from bbox_head and mask_head forward.

loss(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection roi on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• rpn_results_list (List[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

a dictionary of loss components of all stage.

Return type

dict

mask_loss(stage: int, x: Tuple[Tensor], bbox_results: dict, batch_gt_instances: List[InstanceData], rcnn_train_cfg: ConfigDict) → dict

Run forward function and calculate loss for mask head in training.

Parameters

• stage (int) – The current stage in Cascade RoI Head.

• x (tuple[Tensor]) – Tuple of multi-level img features.

• bbox_results (dict) – Results obtained from bbox_loss.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

• (obj (rcnn_train_cfg) – ConfigDict): train_cfg of RCNN.

Returns

Usually returns a dictionary with keys:

• mask_preds (Tensor): Mask prediction.

• loss_mask (dict): A dictionary of mask loss components.

Return type

dict

predict_bbox(x: Tuple[Tensor], batch_img_metas: List[dict], rpn_results_list: List[InstanceData], rcnn_test_cfg: Union[ConfigDict, dict], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the bbox head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• (obj (rcnn_test_cfg) – ConfigDict): test_cfg of R-CNN.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

predict_mask(x: Tuple[Tensor], batch_img_metas: List[dict], results_list: List[InstanceData], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the mask head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• results_list (list[InstanceData]) –

  Detection results of each image. Each item usually contains following keys:

  • scores (Tensor): Classification scores, has a shape (num_instance, )

  • labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

  • bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

  • proposal (Tensor): Bboxes predicted from bbox_head, has a shape (num_instances, 4).

  • topk_inds (Tensor): Topk indices of each image, has shape (num_instances, )

  • attn_feats (Tensor): Intermediate feature get from the last diihead, has shape (num_instances, feature_dimensions)

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[InstanceData]

class mmdet.models.roi_heads.StandardRoIHead(bbox_roi_extractor: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, bbox_head: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, mask_roi_extractor: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, mask_head: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None, shared_head: Optional[Union[ConfigDict, dict]] = None, train_cfg: Optional[Union[ConfigDict, dict]] = None, test_cfg: Optional[Union[ConfigDict, dict]] = None, init_cfg: Optional[Union[ConfigDict, dict, List[Union[ConfigDict, dict]]]] = None)

Simplest base roi head including one bbox head and one mask head.

bbox_loss(x: Tuple[Tensor], sampling_results: List[SamplingResult]) → dict

Perform forward propagation and loss calculation of the bbox head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

Returns

Usually returns a dictionary with keys:

• cls_score (Tensor): Classification scores.

• bbox_pred (Tensor): Box energies / deltas.

• bbox_feats (Tensor): Extract bbox RoI features.

• loss_bbox (dict): A dictionary of bbox loss components.

Return type

dict[str, Tensor]

forward(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: Optional[List[DetDataSample]] = None) → tuple

Network forward process. Usually includes backbone, neck and head forward without any post-processing.

Parameters

• x (List[Tensor]) – Multi-level features that may have different resolutions.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – Each item contains

• corresponding (the meta information of each image and) –

• annotations. –

Returns

tuple: A tuple of features from bbox_head and mask_head forward.

init_assigner_sampler() → None

Initialize assigner and sampler.

init_bbox_head(bbox_roi_extractor: Union[ConfigDict, dict], bbox_head: Union[ConfigDict, dict]) → None

Initialize box head and box roi extractor.

Parameters

• bbox_roi_extractor (dict or ConfigDict) – Config of box roi extractor.

• bbox_head (dict or ConfigDict) – Config of box in box head.

init_mask_head(mask_roi_extractor: Union[ConfigDict, dict], mask_head: Union[ConfigDict, dict]) → None

Initialize mask head and mask roi extractor.

Parameters

• mask_roi_extractor (dict or ConfigDict) – Config of mask roi extractor.

• mask_head (dict or ConfigDict) – Config of mask in mask head.

loss(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample]) → dict

Perform forward propagation and loss calculation of the detection roi on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – List of multi-level img features.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• batch_data_samples (list[DetDataSample]) – The batch data samples. It usually includes information such as gt_instance or gt_panoptic_seg or gt_sem_seg.

Returns

A dictionary of loss components

Return type

dict[str, Tensor]

mask_loss(x: Tuple[Tensor], sampling_results: List[SamplingResult], bbox_feats: Tensor, batch_gt_instances: List[InstanceData]) → dict

Perform forward propagation and loss calculation of the mask head on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Tuple of multi-level img features.

• (list["obj (sampling_results) – SamplingResult]): Sampling results.

• bbox_feats (Tensor) – Extract bbox RoI features.

• batch_gt_instances (list[InstanceData]) – Batch of gt_instance. It usually includes bboxes, labels, and masks attributes.

Returns

Usually returns a dictionary with keys:

• mask_preds (Tensor): Mask prediction.

• mask_feats (Tensor): Extract mask RoI features.

• mask_targets (Tensor): Mask target of each positive proposals in the image.

• loss_mask (dict): A dictionary of mask loss components.

Return type

dict

predict_bbox(x: Tuple[Tensor], batch_img_metas: List[dict], rpn_results_list: List[InstanceData], rcnn_test_cfg: Union[ConfigDict, dict], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the bbox head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• rpn_results_list (list[InstanceData]) – List of region proposals.

• (obj (rcnn_test_cfg) – ConfigDict): test_cfg of R-CNN.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[InstanceData]

predict_mask(x: Tuple[Tensor], batch_img_metas: List[dict], results_list: List[InstanceData], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the mask head and predict detection results on the features of the upstream network.

Parameters

• x (tuple[Tensor]) – Feature maps of all scale level.

• batch_img_metas (list[dict]) – List of image information.

• results_list (list[InstanceData]) – Detection results of each image.

• rescale (bool) – If True, return boxes in original image space. Defaults to False.

Returns

Detection results of each image after the post process. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• masks (Tensor): Has a shape (num_instances, H, W).

Return type

list[InstanceData]

class mmdet.models.roi_heads.TridentRoIHead(num_branch: int, test_branch_idx: int, **kwargs)

Trident roi head.

Parameters

• num_branch (int) – Number of branches in TridentNet.

• test_branch_idx (int) – In inference, all 3 branches will be used if test_branch_idx==-1, otherwise only branch with index test_branch_idx will be used.

merge_trident_bboxes(trident_results: List[InstanceData]) → InstanceData

Merge bbox predictions of each branch.

Parameters

trident_results (List[InstanceData]) – A list of InstanceData predicted from every branch.

Returns

merged InstanceData.

Return type

InstanceData

predict(x: Tuple[Tensor], rpn_results_list: List[InstanceData], batch_data_samples: List[DetDataSample], rescale: bool = False) → List[InstanceData]

Perform forward propagation of the roi head and predict detection results on the features of the upstream network.

• Compute prediction bbox and label per branch.

• Merge predictions of each branch according to scores of bboxes, i.e., bboxes with higher score are kept to give top-k prediction.

Parameters

• x (tuple[Tensor]) – Features from upstream network. Each has shape (N, C, H, W).

• rpn_results_list (list[InstanceData]) – list of region proposals.

• batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• rescale (bool) – Whether to rescale the results to the original image. Defaults to True.

Returns

InstanceData]: Detection results of each image. Each item usually contains following keys.

• scores (Tensor): Classification scores, has a shape (num_instance, )

• labels (Tensor): Labels of bboxes, has a shape (num_instances, ).

• bboxes (Tensor): Has a shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

Return type

list[obj

seg_heads

task_modules

mmdet.models.task_modules.build_assigner(cfg, **default_args)

Builder of box assigner.

mmdet.models.task_modules.build_bbox_coder(cfg, **default_args)

Builder of box coder.

mmdet.models.task_modules.build_iou_calculator(cfg, default_args=None)

Builder of IoU calculator.

mmdet.models.task_modules.build_match_cost(cfg, default_args=None)

Builder of IoU calculator.

mmdet.models.task_modules.build_sampler(cfg, **default_args)

Builder of box sampler.

test_time_augs

class mmdet.models.test_time_augs.DetTTAModel(tta_cfg=None, **kwargs)

Merge augmented detection results, only bboxes corresponding score under flipping and multi-scale resizing can be processed now.

Examples

>>> tta_model = dict(
>>>     type='DetTTAModel',
>>>     tta_cfg=dict(nms=dict(
>>>                     type='nms',
>>>                     iou_threshold=0.5),
>>>                     max_per_img=100))
>>>
>>> tta_pipeline = [
>>>     dict(type='LoadImageFromFile',
>>>          backend_args=None),
>>>     dict(
>>>         type='TestTimeAug',
>>>         transforms=[[
>>>             dict(type='Resize',
>>>                  scale=(1333, 800),
>>>                  keep_ratio=True),
>>>         ], [
>>>             dict(type='RandomFlip', prob=1.),
>>>             dict(type='RandomFlip', prob=0.)
>>>         ], [
>>>             dict(
>>>                 type='PackDetInputs',
>>>                 meta_keys=('img_id', 'img_path', 'ori_shape',
>>>                         'img_shape', 'scale_factor', 'flip',
>>>                         'flip_direction'))
>>>         ]])]

merge_aug_bboxes(aug_bboxes: List[Tensor], aug_scores: List[Tensor], img_metas: List[str]) → Tuple[Tensor, Tensor]

Merge augmented detection bboxes and scores.

Parameters

• aug_bboxes (list[Tensor]) – shape (n, 4*#class)

• aug_scores (list[Tensor] or None) – shape (n, #class)

Returns

bboxes with shape (n,4), where 4 represent (tl_x, tl_y, br_x, br_y) and scores with shape (n,).

Return type

tuple[Tensor]

merge_preds(data_samples_list: List[List[DetDataSample]])

Merge batch predictions of enhanced data.

Parameters

data_samples_list (List[List[DetDataSample]]) – List of predictions of all enhanced data. The outer list indicates images, and the inner list corresponds to the different views of one image. Each element of the inner list is a DetDataSample.

Returns

Merged batch prediction.

Return type

List[DetDataSample]

mmdet.models.test_time_augs.merge_aug_bboxes(aug_bboxes, aug_scores, img_metas, rcnn_test_cfg)

Merge augmented detection bboxes and scores.

Parameters

• aug_bboxes (list[Tensor]) – shape (n, 4*#class)

• aug_scores (list[Tensor] or None) – shape (n, #class)

• img_shapes (list[Tensor]) – shape (3, ).

• rcnn_test_cfg (dict) – rcnn test config.

Returns

(bboxes, scores)

Return type

tuple

mmdet.models.test_time_augs.merge_aug_masks(aug_masks: List[Tensor], img_metas: dict, weights: Optional[Union[list, Tensor]] = None) → Tensor

Merge augmented mask prediction.

Parameters

• aug_masks (list[Tensor]) – each has shape (n, c, h, w).

• img_metas (dict) – Image information.

• weights (list or Tensor) – Weight of each aug_masks, the length should be n.

Returns

has shape (n, c, h, w)

Return type

Tensor

mmdet.models.test_time_augs.merge_aug_proposals(aug_proposals, img_metas, cfg)

Merge augmented proposals (multiscale, flip, etc.)

Parameters

• aug_proposals (list[Tensor]) – proposals from different testing schemes, shape (n, 5). Note that they are not rescaled to the original image size.

• img_metas (list[dict]) – list of image info dict where each dict has: ‘img_shape’, ‘scale_factor’, ‘flip’, and may also contain ‘filename’, ‘ori_shape’, ‘pad_shape’, and ‘img_norm_cfg’. For details on the values of these keys see mmdet/datasets/pipelines/formatting.py:Collect.

• cfg (dict) – rpn test config.

Returns

shape (n, 4), proposals corresponding to original image scale.

Return type

Tensor

mmdet.models.test_time_augs.merge_aug_results(aug_batch_results, aug_batch_img_metas)

Merge augmented detection results, only bboxes corresponding score under flipping and multi-scale resizing can be processed now.

Parameters

• (list[list[[obj (aug_batch_results) –

  InstanceData]]): Detection results of multiple images with different augmentations. The outer list indicate the augmentation . The inter list indicate the batch dimension. Each item usually contains the following keys.

  • scores (Tensor): Classification scores, in shape (num_instance,)

  • labels (Tensor): Labels of bboxes, in shape (num_instances,).

  • bboxes (Tensor): In shape (num_instances, 4), the last dimension 4 arrange as (x1, y1, x2, y2).

• aug_batch_img_metas (list[list[dict]]) – The outer list indicates test-time augs (multiscale, flip, etc.) and the inner list indicates images in a batch. Each dict in the list contains information of an image in the batch.

Returns

InstanceData]): Same with the input aug_results except that all bboxes have been mapped to the original scale.

Return type

batch_results (list[obj

mmdet.models.test_time_augs.merge_aug_scores(aug_scores)

Merge augmented bbox scores.

utils

class mmdet.models.utils.BertEncoderLayer(config: BertConfig, clamp_min_for_underflow: bool = False, clamp_max_for_overflow: bool = False)

A modified version of the BertLayer class from the transformers.models.bert.modeling_bert module.

Parameters

• config (BertConfig) – The configuration object that contains various parameters for the model.

• clamp_min_for_underflow (bool, optional) –

  Whether to clamp the minimum value of the hidden states

  to prevent underflow. Defaults to False.

• clamp_max_for_overflow (bool, optional) – Whether to clamp the maximum value of the hidden states to prevent overflow. Defaults to False.

feed_forward_chunk(attention_output: Tensor) → Tensor

Applies the intermediate and output layers of the BertEncoderLayer to a chunk of the input sequence.

forward(inputs: Dict[str, Dict[str, Tensor]]) → Dict[str, Dict[str, Tensor]]

Applies the BertEncoderLayer to the input features.

class mmdet.models.utils.VLFuse(v_dim: int = 256, l_dim: int = 768, embed_dim: int = 2048, num_heads: int = 8, dropout: float = 0.1, drop_path: float = 0.0, use_checkpoint: bool = False)

Early Fusion Module.

Parameters

• v_dim (int) – Dimension of visual features.

• l_dim (int) – Dimension of language features.

• embed_dim (int) – The embedding dimension for the attention operation.

• num_heads (int) – Number of attention heads.

• dropout (float) – Dropout probability.

• drop_path (float) – Drop path probability.

• use_checkpoint (bool) – Whether to use PyTorch’s checkpoint function.

forward(x: dict) → dict

Forward pass of the VLFuse module.

mmdet.models.utils.align_tensor(inputs: List[Tensor], max_len: Optional[int] = None) → Tensor

Pad each input to max_len, then stack them. If max_len is None, then it is the max size of the first dimension of each input.

https://github.com/jshilong/DDQ/blob/ddq_detr/projects/models/ utils.py#L12

Parameters

• inputs (list[Tensor]) – The tensors to be padded, Each input should have the same shape except the first dimension.

• max_len (int) – Padding target size in the first dimension. Default: None

Returns

Stacked inputs after padding in the first dimension.

Return type

Tensor

mmdet.models.utils.aligned_bilinear(tensor: Tensor, factor: int) → Tensor

Aligned bilinear, used in original implement in CondInst:

https://github.com/aim-uofa/AdelaiDet/blob/ c0b2092ce72442b0f40972f7c6dda8bb52c46d16/adet/utils/comm.py#L23

mmdet.models.utils.center_of_mass(mask, esp=1e-06)

Calculate the centroid coordinates of the mask.

Parameters

• mask (Tensor) – The mask to be calculated, shape (h, w).

• esp (float) – Avoid dividing by zero. Default: 1e-6.

Returns

the coordinates of the center point of the mask.

• center_h (Tensor): the center point of the height.

• center_w (Tensor): the center point of the width.

Return type

tuple[Tensor]

mmdet.models.utils.empty_instances(batch_img_metas: List[dict], device: device, task_type: str, instance_results: Optional[List[InstanceData]] = None, mask_thr_binary: Union[int, float] = 0, box_type: Union[str, type] = 'hbox', use_box_type: bool = False, num_classes: int = 80, score_per_cls: bool = False) → List[InstanceData]

Handle predicted instances when RoI is empty.

Note: If instance_results is not None, it will be modified in place internally, and then return instance_results

Parameters

• batch_img_metas (list[dict]) – List of image information.

• device (torch.device) – Device of tensor.

• task_type (str) – Expected returned task type. it currently supports bbox and mask.

• instance_results (list[InstanceData]) – List of instance results.

• mask_thr_binary (int, float) – mask binarization threshold. Defaults to 0.

• box_type (str or type) – The empty box type. Defaults to hbox.

• use_box_type (bool) – Whether to warp boxes with the box type. Defaults to False.

• num_classes (int) – num_classes of bbox_head. Defaults to 80.

• score_per_cls (bool) – Whether to generate classwise score for the empty instance. score_per_cls will be True when the model needs to produce raw results without nms. Defaults to False.

Returns

Detection results of each image

Return type

list[InstanceData]

mmdet.models.utils.filter_gt_instances(batch_data_samples: List[DetDataSample], score_thr: Optional[float] = None, wh_thr: Optional[tuple] = None)

Filter ground truth (GT) instances by score and/or size.

Parameters

• batch_data_samples (SampleList) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

• score_thr (float) – The score filter threshold.

• wh_thr (tuple) – Minimum width and height of bbox.

Returns

The Data Samples filtered by score and/or size.

Return type

SampleList

mmdet.models.utils.filter_scores_and_topk(scores, score_thr, topk, results=None)

Filter results using score threshold and topk candidates.

Parameters

• scores (Tensor) – The scores, shape (num_bboxes, K).

• score_thr (float) – The score filter threshold.

• topk (int) – The number of topk candidates.

• results (dict or list or Tensor, Optional) – The results to which the filtering rule is to be applied. The shape of each item is (num_bboxes, N).

Returns

Filtered results

• scores (Tensor): The scores after being filtered, shape (num_bboxes_filtered, ).

• labels (Tensor): The class labels, shape (num_bboxes_filtered, ).

• anchor_idxs (Tensor): The anchor indexes, shape (num_bboxes_filtered, ).

• filtered_results (dict or list or Tensor, Optional): The filtered results. The shape of each item is (num_bboxes_filtered, N).

Return type

tuple

mmdet.models.utils.flip_tensor(src_tensor, flip_direction)

Flip tensor base on flip_direction.

Parameters

• src_tensor (Tensor) – input feature map, shape (B, C, H, W).

• flip_direction (str) – The flipping direction. Options are ‘horizontal’, ‘vertical’, ‘diagonal’.

Returns

Flipped tensor.

Return type

out_tensor (Tensor)

mmdet.models.utils.gather_feat(feat, ind, mask=None)

Gather feature according to index.

Parameters

• feat (Tensor) – Target feature map.

• ind (Tensor) – Target coord index.

• mask (Tensor | None) – Mask of feature map. Default: None.

Returns

Gathered feature.

Return type

feat (Tensor)

mmdet.models.utils.gaussian_radius(det_size, min_overlap)

Generate 2D gaussian radius.

This function is modified from the official github repo.

Given min_overlap, radius could computed by a quadratic equation according to Vieta’s formulas.

There are 3 cases for computing gaussian radius, details are following:

• Explanation of figure: lt and br indicates the left-top and bottom-right corner of ground truth box. x indicates the generated corner at the limited position when radius=r.

• Case1: one corner is inside the gt box and the other is outside.

|<   width   >|

lt-+----------+         -
|  |          |         ^
+--x----------+--+
|  |          |  |
|  |          |  |    height
|  | overlap  |  |
|  |          |  |
|  |          |  |      v
+--+---------br--+      -
   |          |  |
   +----------+--x

To ensure IoU of generated box and gt box is larger than min_overlap:

\[\begin{split}\cfrac{(w-r)*(h-r)}{w*h+(w+h)r-r^2} \ge {iou} \quad\Rightarrow\quad {r^2-(w+h)r+\cfrac{1-iou}{1+iou}*w*h} \ge 0 \\ {a} = 1,\quad{b} = {-(w+h)},\quad{c} = {\cfrac{1-iou}{1+iou}*w*h} {r} \le \cfrac{-b-\sqrt{b^2-4*a*c}}{2*a}\end{split}\]

• Case2: both two corners are inside the gt box.

|<   width   >|

lt-+----------+         -
|  |          |         ^
+--x-------+  |
|  |       |  |
|  |overlap|  |       height
|  |       |  |
|  +-------x--+
|          |  |         v
+----------+-br         -

To ensure IoU of generated box and gt box is larger than min_overlap:

\[\begin{split}\cfrac{(w-2*r)*(h-2*r)}{w*h} \ge {iou} \quad\Rightarrow\quad {4r^2-2(w+h)r+(1-iou)*w*h} \ge 0 \\ {a} = 4,\quad {b} = {-2(w+h)},\quad {c} = {(1-iou)*w*h} {r} \le \cfrac{-b-\sqrt{b^2-4*a*c}}{2*a}\end{split}\]

• Case3: both two corners are outside the gt box.

   |<   width   >|

x--+----------------+
|  |                |
+-lt-------------+  |   -
|  |             |  |   ^
|  |             |  |
|  |   overlap   |  | height
|  |             |  |
|  |             |  |   v
|  +------------br--+   -
|                |  |
+----------------+--x

To ensure IoU of generated box and gt box is larger than min_overlap:

\[\begin{split}\cfrac{w*h}{(w+2*r)*(h+2*r)} \ge {iou} \quad\Rightarrow\quad {4*iou*r^2+2*iou*(w+h)r+(iou-1)*w*h} \le 0 \\ {a} = {4*iou},\quad {b} = {2*iou*(w+h)},\quad {c} = {(iou-1)*w*h} \\ {r} \le \cfrac{-b+\sqrt{b^2-4*a*c}}{2*a}\end{split}\]

Parameters

• det_size (list[int]) – Shape of object.

• min_overlap (float) – Min IoU with ground truth for boxes generated by keypoints inside the gaussian kernel.

Returns

Radius of gaussian kernel.

Return type

radius (int)

mmdet.models.utils.gen_gaussian_target(heatmap, center, radius, k=1)

Generate 2D gaussian heatmap.

Parameters

• heatmap (Tensor) – Input heatmap, the gaussian kernel will cover on it and maintain the max value.

• center (list[int]) – Coord of gaussian kernel’s center.

• radius (int) – Radius of gaussian kernel.

• k (int) – Coefficient of gaussian kernel. Default: 1.

Returns

Updated heatmap covered by gaussian kernel.

Return type

out_heatmap (Tensor)

mmdet.models.utils.generate_coordinate(featmap_sizes, device='cuda')

Generate the coordinate.

Parameters

• featmap_sizes (tuple) – The feature to be calculated, of shape (N, C, W, H).

• device (str) – The device where the feature will be put on.

Returns

The coordinate feature, of shape (N, 2, W, H).

Return type

coord_feat (Tensor)

mmdet.models.utils.get_local_maximum(heat, kernel=3)

Extract local maximum pixel with given kernel.

Parameters

• heat (Tensor) – Target heatmap.

• kernel (int) – Kernel size of max pooling. Default: 3.

Returns

A heatmap where local maximum pixels maintain its

own value and other positions are 0.

Return type

heat (Tensor)

mmdet.models.utils.get_topk_from_heatmap(scores, k=20)

Get top k positions from heatmap.

Parameters

• scores (Tensor) – Target heatmap with shape [batch, num_classes, height, width].

• k (int) – Target number. Default: 20.

Returns

Scores, indexes, categories and coords of

topk keypoint. Containing following Tensors:

• topk_scores (Tensor): Max scores of each topk keypoint.

• topk_inds (Tensor): Indexes of each topk keypoint.

• topk_clses (Tensor): Categories of each topk keypoint.

• topk_ys (Tensor): Y-coord of each topk keypoint.

• topk_xs (Tensor): X-coord of each topk keypoint.

Return type

tuple[torch.Tensor]

mmdet.models.utils.get_uncertain_point_coords_with_randomness(mask_preds: Tensor, labels: Tensor, num_points: int, oversample_ratio: float, importance_sample_ratio: float) → Tensor

Get num_points most uncertain points with random points during train.

Sample points in [0, 1] x [0, 1] coordinate space based on their uncertainty. The uncertainties are calculated for each point using ‘get_uncertainty()’ function that takes point’s logit prediction as input.

Parameters

• mask_preds (Tensor) – A tensor of shape (num_rois, num_classes, mask_height, mask_width) for class-specific or class-agnostic prediction.

• labels (Tensor) – The ground truth class for each instance.

• num_points (int) – The number of points to sample.

• oversample_ratio (float) – Oversampling parameter.

• importance_sample_ratio (float) – Ratio of points that are sampled via importnace sampling.

Returns

A tensor of shape (num_rois, num_points, 2)

that contains the coordinates sampled points.

Return type

point_coords (Tensor)

mmdet.models.utils.get_uncertainty(mask_preds: Tensor, labels: Tensor) → Tensor

Estimate uncertainty based on pred logits.

We estimate uncertainty as L1 distance between 0.0 and the logits prediction in ‘mask_preds’ for the foreground class in classes.

Parameters

• mask_preds (Tensor) – mask predication logits, shape (num_rois, num_classes, mask_height, mask_width).

• labels (Tensor) – Either predicted or ground truth label for each predicted mask, of length num_rois.

Returns

Uncertainty scores with the most uncertain

locations having the highest uncertainty score, shape (num_rois, 1, mask_height, mask_width)

Return type

scores (Tensor)

mmdet.models.utils.images_to_levels(target, num_levels)

Convert targets by image to targets by feature level.

[target_img0, target_img1] -> [target_level0, target_level1, …]

mmdet.models.utils.imrenormalize(img: Union[Tensor, ndarray], img_norm_cfg: dict, new_img_norm_cfg: dict) → Union[Tensor, ndarray]

Re-normalize the image.

Parameters

• img (Tensor | ndarray) – Input image. If the input is a Tensor, the shape is (1, C, H, W). If the input is a ndarray, the shape is (H, W, C).

• img_norm_cfg (dict) – Original configuration for the normalization.

• new_img_norm_cfg (dict) – New configuration for the normalization.

Returns

Output image with the same type and shape of the input.

Return type

Tensor | ndarray

mmdet.models.utils.interpolate_as(source, target, mode='bilinear', align_corners=False)

Interpolate the source to the shape of the target.

The source must be a Tensor, but the target can be a Tensor or a np.ndarray with the shape (…, target_h, target_w).

Parameters

• source (Tensor) – A 3D/4D Tensor with the shape (N, H, W) or (N, C, H, W).

• target (Tensor | np.ndarray) – The interpolation target with the shape (…, target_h, target_w).

• mode (str) – Algorithm used for interpolation. The options are the same as those in F.interpolate(). Default: 'bilinear'.

• align_corners (bool) – The same as the argument in F.interpolate().

Returns

The interpolated source Tensor.

Return type

Tensor

mmdet.models.utils.levels_to_images(mlvl_tensor: List[Tensor]) → List[Tensor]

Concat multi-level feature maps by image.

[feature_level0, feature_level1…] -> [feature_image0, feature_image1…] Convert the shape of each element in mlvl_tensor from (N, C, H, W) to (N, H*W , C), then split the element to N elements with shape (H*W, C), and concat elements in same image of all level along first dimension.

Parameters

mlvl_tensor (list[Tensor]) – list of Tensor which collect from corresponding level. Each element is of shape (N, C, H, W)

Returns

A list that contains N tensors and each tensor is

of shape (num_elements, C)

Return type

list[Tensor]

mmdet.models.utils.make_divisible(value, divisor, min_value=None, min_ratio=0.9)

Make divisible function.

This function rounds the channel number to the nearest value that can be divisible by the divisor. It is taken from the original tf repo. It ensures that all layers have a channel number that is divisible by divisor. It can be seen here: https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet.py # noqa

Parameters

• value (int) – The original channel number.

• divisor (int) – The divisor to fully divide the channel number.

• min_value (int) – The minimum value of the output channel. Default: None, means that the minimum value equal to the divisor.

• min_ratio (float) – The minimum ratio of the rounded channel number to the original channel number. Default: 0.9.

Returns

The modified output channel number.

Return type

int

mmdet.models.utils.mask2ndarray(mask)

Convert Mask to ndarray..

:param mask (BitmapMasks or PolygonMasks or: :param torch.Tensor or np.ndarray): The mask to be converted.

Returns

Ndarray mask of shape (n, h, w) that has been converted

Return type

np.ndarray

mmdet.models.utils.multi_apply(func, *args, **kwargs)

Apply function to a list of arguments.

Note

This function applies the func to multiple inputs and map the multiple outputs of the func into different list. Each list contains the same type of outputs corresponding to different inputs.

Parameters

func (Function) – A function that will be applied to a list of arguments

Returns

A tuple containing multiple list, each list contains a kind of returned results by the function

Return type

tuple(list)

mmdet.models.utils.permute_and_flatten(layer: Tensor, N: int, A: int, C: int, H: int, W: int) → Tensor

Permute and then flatten a tensor,

from size (N, A, C, H, W) to (N, H * W * A, C).

Parameters

• layer (Tensor) – Tensor of shape (N, C, H, W).

• N (int) – Batch size.

• A (int) – Number of attention heads.

• C (int) – Number of channels.

• H (int) – Height of feature map.

• W (int) – Width of feature map.

Returns

A Tensor of shape (N, H * W * A, C).

Return type

Tensor

mmdet.models.utils.preprocess_panoptic_gt(gt_labels: Tensor, gt_masks: Tensor, gt_semantic_seg: Tensor, num_things: int, num_stuff: int) → Tuple[Tensor, Tensor]

Preprocess the ground truth for a image.

Parameters

• gt_labels (Tensor) – Ground truth labels of each bbox, with shape (num_gts, ).

• gt_masks (BitmapMasks) – Ground truth masks of each instances of a image, shape (num_gts, h, w).

• gt_semantic_seg (Tensor | None) – Ground truth of semantic segmentation with the shape (1, h, w). [0, num_thing_class - 1] means things, [num_thing_class, num_class-1] means stuff, 255 means VOID. It’s None when training instance segmentation.

Returns

a tuple containing the following targets.

• labels (Tensor): Ground truth class indices for a

  image, with shape (n, ), n is the sum of number of stuff type and number of instance in a image.

• masks (Tensor): Ground truth mask for a image, with

  shape (n, h, w). Contains stuff and things when training panoptic segmentation, and things only when training instance segmentation.

Return type

tuple[Tensor, Tensor]

mmdet.models.utils.relative_coordinate_maps(locations: Tensor, centers: Tensor, strides: Tensor, size_of_interest: int, feat_sizes: Tuple[int]) → Tensor

Generate the relative coordinate maps with feat_stride.

Parameters

• locations (Tensor) – The prior location of mask feature map. It has shape (num_priors, 2).

• centers (Tensor) – The prior points of a object in all feature pyramid. It has shape (num_pos, 2)

• strides (Tensor) – The prior strides of a object in all feature pyramid. It has shape (num_pos, 1)

• size_of_interest (int) – The size of the region used in rel coord.

• feat_sizes (Tuple[int]) – The feature size H and W, which has 2 dims.

Returns

The coordinate feature

of shape (num_pos, 2, H, W).

Return type

rel_coord_feat (Tensor)

mmdet.models.utils.rename_loss_dict(prefix: str, losses: dict) → dict

Rename the key names in loss dict by adding a prefix.

Parameters

• prefix (str) – The prefix for loss components.

• losses (dict) – A dictionary of loss components.

Returns

A dictionary of loss components with prefix.

Return type

dict

mmdet.models.utils.reweight_loss_dict(losses: dict, weight: float) → dict

Reweight losses in the dict by weight.

Parameters

• losses (dict) – A dictionary of loss components.

• weight (float) – Weight for loss components.

Returns

A dictionary of weighted loss components.

Return type

dict

mmdet.models.utils.select_single_mlvl(mlvl_tensors, batch_id, detach=True)

Extract a multi-scale single image tensor from a multi-scale batch tensor based on batch index.

Note: The default value of detach is True, because the proposal gradient needs to be detached during the training of the two-stage model. E.g Cascade Mask R-CNN.

Parameters

• mlvl_tensors (list[Tensor]) – Batch tensor for all scale levels, each is a 4D-tensor.

• batch_id (int) – Batch index.

• detach (bool) – Whether detach gradient. Default True.

Returns

Multi-scale single image tensor.

Return type

list[Tensor]

mmdet.models.utils.transpose_and_gather_feat(feat, ind)

Transpose and gather feature according to index.

Parameters

• feat (Tensor) – Target feature map.

• ind (Tensor) – Target coord index.

Returns

Transposed and gathered feature.

Return type

feat (Tensor)

mmdet.models.utils.unfold_wo_center(x, kernel_size: int, dilation: int) → Tensor

unfold_wo_center, used in original implement in BoxInst:

https://github.com/aim-uofa/AdelaiDet/blob/ 4a3a1f7372c35b48ebf5f6adc59f135a0fa28d60/ adet/modeling/condinst/condinst.py#L53

mmdet.models.utils.unmap(data, count, inds, fill=0)

Unmap a subset of item (data) back to the original set of items (of size count)

mmdet.models.utils.unpack_gt_instances(batch_data_samples: List[DetDataSample]) → tuple

Unpack gt_instances, gt_instances_ignore and img_metas based on batch_data_samples

Parameters

batch_data_samples (List[DetDataSample]) – The Data Samples. It usually includes information such as gt_instance, gt_panoptic_seg and gt_sem_seg.

Returns

• batch_gt_instances (list[InstanceData]): Batch of

  gt_instance. It usually includes bboxes and labels attributes.

• batch_gt_instances_ignore (list[InstanceData]):

  Batch of gt_instances_ignore. It includes bboxes attribute data that is ignored during training and testing. Defaults to None.

• batch_img_metas (list[dict]): Meta information of each image,

  e.g., image size, scaling factor, etc.

Return type

tuple

mmdet.models.utils.weighted_boxes_fusion(bboxes_list: list, scores_list: list, labels_list: list, weights: Optional[list] = None, iou_thr: float = 0.55, skip_box_thr: float = 0.0, conf_type: str = 'avg', allows_overflow: bool = False) → Tuple[Tensor, Tensor, Tensor]

weighted boxes fusion <https://arxiv.org/abs/1910.13302> is a method for fusing predictions from different object detection models, which utilizes confidence scores of all proposed bounding boxes to construct averaged boxes.

Parameters

• bboxes_list (list) – list of boxes predictions from each model, each box is 4 numbers.

• scores_list (list) – list of scores for each model

• labels_list (list) – list of labels for each model

• weights – list of weights for each model. Default: None, which means weight == 1 for each model

• iou_thr – IoU value for boxes to be a match

• skip_box_thr – exclude boxes with score lower than this variable.

• conf_type –

  how to calculate confidence in weighted boxes. ‘avg’: average value, ‘max’: maximum value, ‘box_and_model_avg’: box and model wise hybrid weighted average, ‘absent_model_aware_avg’: weighted average that takes into

  account the absent model.

• allows_overflow – false if we want confidence score not exceed 1.0.

Returns

boxes coordinates (Order of boxes: x1, y1, x2, y2). scores(Tensor): confidence scores labels(Tensor): boxes labels

Return type

bboxes(Tensor)

mmdet.structures

structures

class mmdet.structures.DetDataSample(*, metainfo: Optional[dict] = None, **kwargs)

A data structure interface of MMDetection. They are used as interfaces between different components.

The attributes in DetDataSample are divided into several parts:

• ``proposals``(InstanceData): Region proposals used in two-stage

  detectors.

• ``gt_instances``(InstanceData): Ground truth of instance annotations.

• ``pred_instances``(InstanceData): Instances of detection predictions.

• ``pred_track_instances``(InstanceData): Instances of tracking

  predictions.

• ``ignored_instances``(InstanceData): Instances to be ignored during

  training/testing.

• ``gt_panoptic_seg``(PixelData): Ground truth of panoptic

  segmentation.

• ``pred_panoptic_seg``(PixelData): Prediction of panoptic

  segmentation.

• ``gt_sem_seg``(PixelData): Ground truth of semantic segmentation.

• ``pred_sem_seg``(PixelData): Prediction of semantic segmentation.

Examples

>>> import torch
>>> import numpy as np
>>> from mmengine.structures import InstanceData
>>> from mmdet.structures import DetDataSample

>>> data_sample = DetDataSample()
>>> img_meta = dict(img_shape=(800, 1196),
...                 pad_shape=(800, 1216))
>>> gt_instances = InstanceData(metainfo=img_meta)
>>> gt_instances.bboxes = torch.rand((5, 4))
>>> gt_instances.labels = torch.rand((5,))
>>> data_sample.gt_instances = gt_instances
>>> assert 'img_shape' in data_sample.gt_instances.metainfo_keys()
>>> len(data_sample.gt_instances)
5
>>> print(data_sample)

<DetDataSample(

META INFORMATION

DATA FIELDS gt_instances: <InstanceData(

META INFORMATION pad_shape: (800, 1216) img_shape: (800, 1196)

DATA FIELDS labels: tensor([0.8533, 0.1550, 0.5433, 0.7294, 0.5098]) bboxes: tensor([[9.7725e-01, 5.8417e-01, 1.7269e-01, 6.5694e-01],

[1.7894e-01, 5.1780e-01, 7.0590e-01, 4.8589e-01], [7.0392e-01, 6.6770e-01, 1.7520e-01, 1.4267e-01], [2.2411e-01, 5.1962e-01, 9.6953e-01, 6.6994e-01], [4.1338e-01, 2.1165e-01, 2.7239e-04, 6.8477e-01]])

) at 0x7f21fb1b9190>

) at 0x7f21fb1b9880>

>>> pred_instances = InstanceData(metainfo=img_meta)
>>> pred_instances.bboxes = torch.rand((5, 4))
>>> pred_instances.scores = torch.rand((5,))
>>> data_sample = DetDataSample(pred_instances=pred_instances)
>>> assert 'pred_instances' in data_sample

>>> pred_track_instances = InstanceData(metainfo=img_meta)
>>> pred_track_instances.bboxes = torch.rand((5, 4))
>>> pred_track_instances.scores = torch.rand((5,))
>>> data_sample = DetDataSample(
...    pred_track_instances=pred_track_instances)
>>> assert 'pred_track_instances' in data_sample

>>> data_sample = DetDataSample()
>>> gt_instances_data = dict(
...                        bboxes=torch.rand(2, 4),
...                        labels=torch.rand(2),
...                        masks=np.random.rand(2, 2, 2))
>>> gt_instances = InstanceData(**gt_instances_data)
>>> data_sample.gt_instances = gt_instances
>>> assert 'gt_instances' in data_sample
>>> assert 'masks' in data_sample.gt_instances

>>> data_sample = DetDataSample()
>>> gt_panoptic_seg_data = dict(panoptic_seg=torch.rand(2, 4))
>>> gt_panoptic_seg = PixelData(**gt_panoptic_seg_data)
>>> data_sample.gt_panoptic_seg = gt_panoptic_seg
>>> print(data_sample)

<DetDataSample(

META INFORMATION

DATA FIELDS _gt_panoptic_seg: <BaseDataElement(

META INFORMATION

DATA FIELDS panoptic_seg: tensor([[0.7586, 0.1262, 0.2892, 0.9341],

[0.3200, 0.7448, 0.1052, 0.5371]])

) at 0x7f66c2bb7730>

gt_panoptic_seg: <BaseDataElement(

META INFORMATION

DATA FIELDS panoptic_seg: tensor([[0.7586, 0.1262, 0.2892, 0.9341],

[0.3200, 0.7448, 0.1052, 0.5371]])

) at 0x7f66c2bb7730>

) at 0x7f66c2bb7280> >>> data_sample = DetDataSample() >>> gt_segm_seg_data = dict(segm_seg=torch.rand(2, 2, 2)) >>> gt_segm_seg = PixelData(**gt_segm_seg_data) >>> data_sample.gt_segm_seg = gt_segm_seg >>> assert ‘gt_segm_seg’ in data_sample >>> assert ‘segm_seg’ in data_sample.gt_segm_seg

class mmdet.structures.ReIDDataSample(*, metainfo: Optional[dict] = None, **kwargs)

A data structure interface of ReID task.

It’s used as interfaces between different components.

Meta field:

img_shape (Tuple): The shape of the corresponding input image.

Used for visualization.

ori_shape (Tuple): The original shape of the corresponding image.

Used for visualization.

num_classes (int): The number of all categories.

Used for label format conversion.

Data field:

gt_label (LabelData): The ground truth label. pred_label (LabelData): The predicted label. scores (torch.Tensor): The outputs of model.

set_gt_label(value: Union[ndarray, Tensor, Sequence[Number], Number]) → ReIDDataSample

Set label of gt_label.

set_gt_score(value: Tensor) → ReIDDataSample

Set score of gt_label.

class mmdet.structures.TrackDataSample(*, metainfo: Optional[dict] = None, **kwargs)

A data structure interface of tracking task in MMDetection. It is used as interfaces between different components.

This data structure can be viewd as a wrapper of multiple DetDataSample to some extent. Specifically, it only contains a property: video_data_samples which is a list of DetDataSample, each of which corresponds to a single frame. If you want to get the property of a single frame, you must first get the corresponding DetDataSample by indexing and then get the property of the frame, such as gt_instances, pred_instances and so on. As for metainfo, it differs from DetDataSample in that each value corresponds to the metainfo key is a list where each element corresponds to information of a single frame.

Examples

>>> import torch
>>> from mmengine.structures import InstanceData
>>> from mmdet.structures import DetDataSample, TrackDataSample
>>> track_data_sample = TrackDataSample()
>>> # set the 1st frame
>>> frame1_data_sample = DetDataSample(metainfo=dict(
...         img_shape=(100, 100), frame_id=0))
>>> frame1_gt_instances = InstanceData()
>>> frame1_gt_instances.bbox = torch.zeros([2, 4])
>>> frame1_data_sample.gt_instances = frame1_gt_instances
>>> # set the 2nd frame
>>> frame2_data_sample = DetDataSample(metainfo=dict(
...         img_shape=(100, 100), frame_id=1))
>>> frame2_gt_instances = InstanceData()
>>> frame2_gt_instances.bbox = torch.ones([3, 4])
>>> frame2_data_sample.gt_instances = frame2_gt_instances
>>> track_data_sample.video_data_samples = [frame1_data_sample,
...                                         frame2_data_sample]
>>> # set metainfo for track_data_sample
>>> track_data_sample.set_metainfo(dict(key_frames_inds=[0]))
>>> track_data_sample.set_metainfo(dict(ref_frames_inds=[1]))
>>> print(track_data_sample)
<TrackDataSample(

META INFORMATION key_frames_inds: [0] ref_frames_inds: [1]

DATA FIELDS video_data_samples: [<DetDataSample(

META INFORMATION img_shape: (100, 100)

DATA FIELDS gt_instances: <InstanceData(

META INFORMATION

DATA FIELDS bbox: tensor([[0., 0., 0., 0.],

[0., 0., 0., 0.]])

) at 0x7f639320dcd0>

) at 0x7f64bd223340>, <DetDataSample(

META INFORMATION img_shape: (100, 100)

DATA FIELDS gt_instances: <InstanceData(

META INFORMATION

DATA FIELDS bbox: tensor([[1., 1., 1., 1.],

[1., 1., 1., 1.], [1., 1., 1., 1.]])

) at 0x7f64bd128b20>

) at 0x7f64bd1346d0>]

) at 0x7f64bd2237f0> >>> print(len(track_data_sample)) 2 >>> key_data_sample = track_data_sample.get_key_frames() >>> print(key_data_sample[0].frame_id) 0 >>> ref_data_sample = track_data_sample.get_ref_frames() >>> print(ref_data_sample[0].frame_id) 1 >>> frame1_data_sample = track_data_sample[0] >>> print(frame1_data_sample.gt_instances.bbox) tensor([[0., 0., 0., 0.],

[0., 0., 0., 0.]])

>>> # Tensor-like methods
>>> cuda_track_data_sample = track_data_sample.to('cuda')
>>> cuda_track_data_sample = track_data_sample.cuda()
>>> cpu_track_data_sample = track_data_sample.cpu()
>>> cpu_track_data_sample = track_data_sample.to('cpu')
>>> fp16_instances = cuda_track_data_sample.to(
...     device=None, dtype=torch.float16, non_blocking=False,
...     copy=False, memory_format=torch.preserve_format)

clone() → BaseDataElement

Deep copy the current data element.

Returns

The copy of current data element.

Return type

BaseDataElement

cpu() → BaseDataElement

Convert all tensors to CPU in data.

cuda() → BaseDataElement

Convert all tensors to GPU in data.

detach() → BaseDataElement

Detach all tensors in data.

npu() → BaseDataElement

Convert all tensors to NPU in data.

numpy() → BaseDataElement

Convert all tensors to np.ndarray in data.

to(*args, **kwargs) → BaseDataElement

Apply same name function to all tensors in data_fields.

to_tensor() → BaseDataElement

Convert all np.ndarray to tensor in data.

bbox

class mmdet.structures.bbox.BaseBoxes(data: Union[Tensor, ndarray, Sequence], dtype: Optional[dtype] = None, device: Optional[Union[str, device]] = None, clone: bool = True)

The base class for 2D box types.

The functions of BaseBoxes lie in three fields:

• Verify the boxes shape.

• Support tensor-like operations.

• Define abstract functions for 2D boxes.

In __init__ , BaseBoxes verifies the validity of the data shape w.r.t box_dim. The tensor with the dimension >= 2 and the length of the last dimension being box_dim will be regarded as valid. BaseBoxes will restore them at the field tensor. It’s necessary to override box_dim in subclass to guarantee the data shape is correct.

There are many basic tensor-like functions implemented in BaseBoxes. In most cases, users can operate BaseBoxes instance like a normal tensor. To protect the validity of data shape, All tensor-like functions cannot modify the last dimension of self.tensor.

When creating a new box type, users need to inherit from BaseBoxes and override abstract methods and specify the box_dim. Then, register the new box type by using the decorator register_box_type.

Parameters

• data (Tensor or np.ndarray or Sequence) – The box data with shape (…, box_dim).

• dtype (torch.dtype, Optional) – data type of boxes. Defaults to None.

• device (str or torch.device, Optional) – device of boxes. Default to None.

• clone (bool) – Whether clone boxes or not. Defaults to True.

abstract property areas: Tensor

Return a tensor representing the areas of boxes.

classmethod cat(box_list: Sequence[T], dim: int = 0) → T

Cancatenates a box instance list into one single box instance. Similar to torch.cat.

Parameters

• box_list (Sequence[T]) – A sequence of box instances.

• dim (int) – The dimension over which the box are concatenated. Defaults to 0.

Returns

Concatenated box instance.

Return type

T

abstract property centers: Tensor

Return a tensor representing the centers of boxes.

chunk(chunks: int, dim: int = 0) → List[T]

Reload chunk from self.tensor.

abstract clip_(img_shape: Tuple[int, int]) → None

Clip boxes according to the image shape in-place.

Parameters

img_shape (Tuple[int, int]) – A tuple of image height and width.

clone() → T

Reload clone from self.tensor.

convert_to(dst_type: Union[str, type]) → BaseBoxes

Convert self to another box type.

Parameters

dst_type (str or type) – destination box type.

Returns

destination box type object .

Return type

BaseBoxes

cpu() → T

Reload cpu from self.tensor.

cuda(*args, **kwargs) → T

Reload cuda from self.tensor.

detach() → T

Reload detach from self.tensor.

property device: device

Reload device from self.tensor.

dim() → int

Reload dim from self.tensor.

property dtype: dtype

Reload dtype from self.tensor.

empty_boxes(dtype: Optional[dtype] = None, device: Optional[Union[str, device]] = None) → T

Create empty box.

Parameters

• dtype (torch.dtype, Optional) – data type of boxes.

• device (str or torch.device, Optional) – device of boxes.

Returns

empty boxes with shape of (0, box_dim).

Return type

T

expand(*sizes: Tuple[int]) → T

Reload expand from self.tensor.

fake_boxes(sizes: Tuple[int], fill: float = 0, dtype: Optional[dtype] = None, device: Optional[Union[str, device]] = None) → T

Create fake boxes with specific sizes and fill values.

Parameters

• sizes (Tuple[int]) – The size of fake boxes. The last value must be equal with self.box_dim.

• fill (float) – filling value. Defaults to 0.

• dtype (torch.dtype, Optional) – data type of boxes.

• device (str or torch.device, Optional) – device of boxes.

Returns

Fake boxes with shape of sizes.

Return type

T

abstract find_inside_points(points: Tensor, is_aligned: bool = False) → BoolTensor

Find inside box points. Boxes dimension must be 2.

Parameters

• points (Tensor) – Points coordinates. Has shape of (m, 2).

• is_aligned (bool) – Whether points has been aligned with boxes or not. If True, the length of boxes and points should be the same. Defaults to False.

Returns

A BoolTensor indicating whether a point is inside boxes. Assuming the boxes has shape of (n, box_dim), if is_aligned is False. The index has shape of (m, n). If is_aligned is True, m should be equal to n and the index has shape of (m, ).

Return type

BoolTensor

flatten(start_dim: int = 0, end_dim: int = -2) → T

Reload flatten from self.tensor.

abstract flip_(img_shape: Tuple[int, int], direction: str = 'horizontal') → None

Flip boxes horizontally or vertically in-place.

Parameters

• img_shape (Tuple[int, int]) – A tuple of image height and width.

• direction (str) – Flip direction, options are “horizontal”, “vertical” and “diagonal”. Defaults to “horizontal”

abstract static from_instance_masks(masks: Union[BitmapMasks, PolygonMasks]) → BaseBoxes

Create boxes from instance masks.

Parameters

masks (BitmapMasks or PolygonMasks) – BitmapMasks or PolygonMasks instance with length of n.

Returns

Converted boxes with shape of (n, box_dim).

Return type

BaseBoxes

abstract property heights: Tensor

Return a tensor representing the heights of boxes.

abstract is_inside(img_shape: Tuple[int, int], all_inside: bool = False, allowed_border: int = 0) → BoolTensor

Find boxes inside the image.

Parameters

• img_shape (Tuple[int, int]) – A tuple of image height and width.

• all_inside (bool) – Whether the boxes are all inside the image or part inside the image. Defaults to False.

• allowed_border (int) – Boxes that extend beyond the image shape boundary by more than allowed_border are considered “outside” Defaults to 0.

Returns

A BoolTensor indicating whether the box is inside the image. Assuming the original boxes have shape (m, n, box_dim), the output has shape (m, n).

Return type

BoolTensor

new_empty(*args, **kwargs) → Tensor

Reload new_empty from self.tensor.

new_full(*args, **kwargs) → Tensor

Reload new_full from self.tensor.

new_ones(*args, **kwargs) → Tensor

Reload new_ones from self.tensor.

new_tensor(*args, **kwargs) → Tensor

Reload new_tensor from self.tensor.

new_zeros(*args, **kwargs) → Tensor

Reload new_zeros from self.tensor.

numel() → int

Reload numel from self.tensor.

numpy() → ndarray

Reload numpy from self.tensor.

abstract static overlaps(boxes1: BaseBoxes, boxes2: BaseBoxes, mode: str = 'iou', is_aligned: bool = False, eps: float = 1e-06) → Tensor

Calculate overlap between two set of boxes with their types converted to the present box type.

Parameters

• boxes1 (BaseBoxes) – BaseBoxes with shape of (m, box_dim) or empty.

• boxes2 (BaseBoxes) – BaseBoxes with shape of (n, box_dim) or empty.

• mode (str) – “iou” (intersection over union), “iof” (intersection over foreground). Defaults to “iou”.

• is_aligned (bool) – If True, then m and n must be equal. Defaults to False.

• eps (float) – A value added to the denominator for numerical stability. Defaults to 1e-6.

Returns

shape (m, n) if is_aligned is False else shape (m,)

Return type

Tensor

permute(*dims: Tuple[int]) → T

Reload permute from self.tensor.

abstract project_(homography_matrix: Union[Tensor, ndarray]) → None

Geometric transformat boxes in-place.

Parameters

homography_matrix (Tensor or np.ndarray]) – Shape (3, 3) for geometric transformation.

repeat(*sizes: Tuple[int]) → T

Reload repeat from self.tensor.

abstract rescale_(scale_factor: Tuple[float, float]) → None

Rescale boxes w.r.t. rescale_factor in-place.

Note

Both rescale_ and resize_ will enlarge or shrink boxes w.r.t scale_facotr. The difference is that resize_ only changes the width and the height of boxes, but rescale_ also rescales the box centers simultaneously.

Parameters

scale_factor (Tuple[float, float]) – factors for scaling boxes. The length should be 2.

reshape(*shape: Tuple[int]) → T

Reload reshape from self.tensor.

abstract resize_(scale_factor: Tuple[float, float]) → None

Resize the box width and height w.r.t scale_factor in-place.

Note

Both rescale_ and resize_ will enlarge or shrink boxes w.r.t scale_facotr. The difference is that resize_ only changes the width and the height of boxes, but rescale_ also rescales the box centers simultaneously.

Parameters

scale_factor (Tuple[float, float]) – factors for scaling box shapes. The length should be 2.

abstract rotate_(center: Tuple[float, float], angle: float) → None

Rotate all boxes in-place.

Parameters

• center (Tuple[float, float]) – Rotation origin.

• angle (float) – Rotation angle represented in degrees. Positive values mean clockwise rotation.

size(dim: Optional[int] = None) → Union[int, Size]

Reload new_zeros from self.tensor.

split(split_size_or_sections: Union[int, Sequence[int]], dim: int = 0) → List[T]

Reload split from self.tensor.

squeeze(dim: Optional[int] = None) → T

Reload squeeze from self.tensor.

classmethod stack(box_list: Sequence[T], dim: int = 0) → T

Concatenates a sequence of tensors along a new dimension. Similar to torch.stack.

Parameters

• box_list (Sequence[T]) – A sequence of box instances.

• dim (int) – Dimension to insert. Defaults to 0.

Returns

Concatenated box instance.

Return type

T

to(*args, **kwargs) → T

Reload to from self.tensor.

abstract translate_(distances: Tuple[float, float]) → None

Translate boxes in-place.

Parameters

distances (Tuple[float, float]) – translate distances. The first is horizontal distance and the second is vertical distance.

transpose(dim0: int, dim1: int) → T

Reload transpose from self.tensor.

unbind(dim: int = 0) → T

Reload unbind from self.tensor.

unsqueeze(dim: int) → T

Reload unsqueeze from self.tensor.

view(*shape: Tuple[int]) → T

Reload view from self.tensor.

abstract property widths: Tensor

Return a tensor representing the widths of boxes.

class mmdet.structures.bbox.HorizontalBoxes(data: Union[Tensor, ndarray], dtype: Optional[dtype] = None, device: Optional[Union[str, device]] = None, clone: bool = True, in_mode: Optional[str] = None)

The horizontal box class used in MMDetection by default.

The box_dim of HorizontalBoxes is 4, which means the length of the last dimension of the data should be 4. Two modes of box data are supported in HorizontalBoxes:

• ‘xyxy’: Each row of data indicates (x1, y1, x2, y2), which are the coordinates of the left-top and right-bottom points.

• ‘cxcywh’: Each row of data indicates (x, y, w, h), where (x, y) are the coordinates of the box centers and (w, h) are the width and height.

HorizontalBoxes only restores ‘xyxy’ mode of data. If the the data is in ‘cxcywh’ mode, users need to input in_mode='cxcywh' and The code will convert the ‘cxcywh’ data to ‘xyxy’ automatically.

Parameters

• data (Tensor or np.ndarray or Sequence) – The box data with shape of (…, 4).

• dtype (torch.dtype, Optional) – data type of boxes. Defaults to None.

• device (str or torch.device, Optional) – device of boxes. Default to None.

• clone (bool) – Whether clone boxes or not. Defaults to True.

• mode (str, Optional) – the mode of boxes. If it is ‘cxcywh’, the data will be converted to ‘xyxy’ mode. Defaults to None.

property areas: Tensor

Return a tensor representing the areas of boxes.

property centers: Tensor

Return a tensor representing the centers of boxes.

clip_(img_shape: Tuple[int, int]) → None

Clip boxes according to the image shape in-place.

Parameters

img_shape (Tuple[int, int]) – A tuple of image height and width.

static corner2hbox(corners: Tensor) → Tensor

Convert box coordinates from corners ((x1, y1), (x2, y1), (x1, y2), (x2, y2)) to (x1, y1, x2, y2).

Parameters

corners (Tensor) – Corner tensor with shape of (…, 4, 2).

Returns

Horizontal box tensor with shape of (…, 4).

Return type

Tensor

create_masks(img_shape: Tuple[int, int]) → BitmapMasks

Parameters

img_shape (Tuple[int, int]) – A tuple of image height and width.

Returns

Converted masks

Return type

BitmapMasks

property cxcywh: Tensor

Return a tensor representing the cxcywh boxes.

static cxcywh_to_xyxy(boxes: Tensor) → Tensor

Convert box coordinates from (cx, cy, w, h) to (x1, y1, x2, y2).

Parameters

boxes (Tensor) – cxcywh boxes tensor with shape of (…, 4).

Returns

xyxy boxes tensor with shape of (…, 4).

Return type

Tensor

find_inside_points(points: Tensor, is_aligned: bool = False) → BoolTensor

Find inside box points. Boxes dimension must be 2.

Parameters

• points (Tensor) – Points coordinates. Has shape of (m, 2).

• is_aligned (bool) – Whether points has been aligned with boxes or not. If True, the length of boxes and points should be the same. Defaults to False.

Returns

A BoolTensor indicating whether a point is inside boxes. Assuming the boxes has shape of (n, 4), if is_aligned is False. The index has shape of (m, n). If is_aligned is True, m should be equal to n and the index has shape of (m, ).

Return type

BoolTensor

flip_(img_shape: Tuple[int, int], direction: str = 'horizontal') → None

Flip boxes horizontally or vertically in-place.

Parameters

• img_shape (Tuple[int, int]) – A tuple of image height and width.

• direction (str) – Flip direction, options are “horizontal”, “vertical” and “diagonal”. Defaults to “horizontal”

static from_instance_masks(masks: Union[BitmapMasks, PolygonMasks]) → HorizontalBoxes

Create horizontal boxes from instance masks.

Parameters

masks (BitmapMasks or PolygonMasks) – BitmapMasks or PolygonMasks instance with length of n.

Returns

Converted boxes with shape of (n, 4).

Return type

HorizontalBoxes

static hbox2corner(boxes: Tensor) → Tensor

Convert box coordinates from (x1, y1, x2, y2) to corners ((x1, y1), (x2, y1), (x1, y2), (x2, y2)).

Parameters

boxes (Tensor) – Horizontal box tensor with shape of (…, 4).

Returns

Corner tensor with shape of (…, 4, 2).

Return type

Tensor

property heights: Tensor

Return a tensor representing the heights of boxes.

is_inside(img_shape: Tuple[int, int], all_inside: bool = False, allowed_border: int = 0) → BoolTensor

Find boxes inside the image.

Parameters

• img_shape (Tuple[int, int]) – A tuple of image height and width.

• all_inside (bool) – Whether the boxes are all inside the image or part inside the image. Defaults to False.

• allowed_border (int) – Boxes that extend beyond the image shape boundary by more than allowed_border are considered “outside” Defaults to 0.

Returns

A BoolTensor indicating whether the box is inside the image. Assuming the original boxes have shape (m, n, 4), the output has shape (m, n).

Return type

BoolTensor

static overlaps(boxes1: BaseBoxes, boxes2: BaseBoxes, mode: str = 'iou', is_aligned: bool = False, eps: float = 1e-06) → Tensor

Calculate overlap between two set of boxes with their types converted to HorizontalBoxes.

Parameters

• boxes1 (BaseBoxes) – BaseBoxes with shape of (m, box_dim) or empty.

• boxes2 (BaseBoxes) – BaseBoxes with shape of (n, box_dim) or empty.

• mode (str) – “iou” (intersection over union), “iof” (intersection over foreground). Defaults to “iou”.

• is_aligned (bool) – If True, then m and n must be equal. Defaults to False.

• eps (float) – A value added to the denominator for numerical stability. Defaults to 1e-6.

Returns

shape (m, n) if is_aligned is False else shape (m,)

Return type

Tensor

project_(homography_matrix: Union[Tensor, ndarray]) → None

Geometric transformat boxes in-place.

Parameters

homography_matrix (Tensor or np.ndarray]) – Shape (3, 3) for geometric transformation.

rescale_(scale_factor: Tuple[float, float]) → None

Rescale boxes w.r.t. rescale_factor in-place.

Note

Both rescale_ and resize_ will enlarge or shrink boxes w.r.t scale_facotr. The difference is that resize_ only changes the width and the height of boxes, but rescale_ also rescales the box centers simultaneously.

Parameters

scale_factor (Tuple[float, float]) – factors for scaling boxes. The length should be 2.

resize_(scale_factor: Tuple[float, float]) → None

Resize the box width and height w.r.t scale_factor in-place.

Note

Both rescale_ and resize_ will enlarge or shrink boxes w.r.t scale_facotr. The difference is that resize_ only changes the width and the height of boxes, but rescale_ also rescales the box centers simultaneously.

Parameters

scale_factor (Tuple[float, float]) – factors for scaling box shapes. The length should be 2.

rotate_(center: Tuple[float, float], angle: float) → None

Rotate all boxes in-place.

Parameters

• center (Tuple[float, float]) – Rotation origin.

• angle (float) – Rotation angle represented in degrees. Positive values mean clockwise rotation.

translate_(distances: Tuple[float, float]) → None

Translate boxes in-place.

Parameters

distances (Tuple[float, float]) – translate distances. The first is horizontal distance and the second is vertical distance.

property widths: Tensor

Return a tensor representing the widths of boxes.

static xyxy_to_cxcywh(boxes: Tensor) → Tensor

Convert box coordinates from (x1, y1, x2, y2) to (cx, cy, w, h).

Parameters

boxes (Tensor) – xyxy boxes tensor with shape of (…, 4).

Returns

cxcywh boxes tensor with shape of (…, 4).

Return type

Tensor

mmdet.structures.bbox.autocast_box_type(dst_box_type='hbox') → Callable

A decorator which automatically casts results[‘gt_bboxes’] to the destination box type.

It commenly used in mmdet.datasets.transforms to make the transforms up- compatible with the np.ndarray type of results[‘gt_bboxes’].

The speed of processing of np.ndarray and BaseBoxes data are the same:

• np.ndarray: 0.0509 img/s

• BaseBoxes: 0.0551 img/s

Parameters

dst_box_type (str) – Destination box type.

mmdet.structures.bbox.bbox2corner(bboxes: Tensor) → Tensor

Convert bbox coordinates from (x1, y1, x2, y2) to corners ((x1, y1), (x2, y1), (x1, y2), (x2, y2)).

Parameters

bboxes (Tensor) – Shape (n, 4) for bboxes.

Returns

Shape (n*4, 2) for corners.

Return type

Tensor

mmdet.structures.bbox.bbox2distance(points: Tensor, bbox: Tensor, max_dis: Optional[float] = None, eps: float = 0.1) → Tensor

Decode bounding box based on distances.

Parameters

• points (Tensor) – Shape (n, 2) or (b, n, 2), [x, y].

• bbox (Tensor) – Shape (n, 4) or (b, n, 4), “xyxy” format

• max_dis (float, optional) – Upper bound of the distance.

• eps (float) – a small value to ensure target < max_dis, instead <=

Returns

Decoded distances.

Return type

Tensor

mmdet.structures.bbox.bbox2result(bboxes: Union[Tensor, ndarray], labels: Union[Tensor, ndarray], num_classes: int) → List[ndarray]

Convert detection results to a list of numpy arrays.

Parameters

• bboxes (Tensor | np.ndarray) – shape (n, 5)

• labels (Tensor | np.ndarray) – shape (n, )

• num_classes (int) – class number, including background class

Returns

bbox results of each class

Return type

List(np.ndarray])

mmdet.structures.bbox.bbox2roi(bbox_list: List[Union[Tensor, BaseBoxes]]) → Tensor

Convert a list of bboxes to roi format.

Parameters

bbox_list (List[Union[Tensor, BaseBoxes]) – a list of bboxes corresponding to a batch of images.

Returns

shape (n, box_dim + 1), where box_dim depends on the different box types. For example, If the box type in bbox_list is HorizontalBoxes, the output shape is (n, 5). Each row of data indicates [batch_ind, x1, y1, x2, y2].

Return type

Tensor

mmdet.structures.bbox.bbox_cxcyah_to_xyxy(bboxes: Tensor) → Tensor

Convert bbox coordinates from (cx, cy, ratio, h) to (x1, y1, x2, y2).

Parameters

bbox (Tensor) – Shape (n, 4) for bboxes.

Returns

Converted bboxes.

Return type

Tensor

mmdet.structures.bbox.bbox_cxcywh_to_xyxy(bbox: Tensor) → Tensor

Convert bbox coordinates from (cx, cy, w, h) to (x1, y1, x2, y2).

Parameters

bbox (Tensor) – Shape (n, 4) for bboxes.

Returns

Converted bboxes.

Return type

Tensor

mmdet.structures.bbox.bbox_flip(bboxes: Tensor, img_shape: Tuple[int], direction: str = 'horizontal') → Tensor

Flip bboxes horizontally or vertically.

Parameters

• bboxes (Tensor) – Shape (…, 4*k)

• img_shape (Tuple[int]) – Image shape.

• direction (str) – Flip direction, options are “horizontal”, “vertical”, “diagonal”. Default: “horizontal”

Returns

Flipped bboxes.

Return type

Tensor

mmdet.structures.bbox.bbox_mapping(bboxes: Tensor, img_shape: Tuple[int], scale_factor: Union[float, Tuple[float]], flip: bool, flip_direction: str = 'horizontal') → Tensor

Map bboxes from the original image scale to testing scale.

mmdet.structures.bbox.bbox_mapping_back(bboxes: Tensor, img_shape: Tuple[int], scale_factor: Union[float, Tuple[float]], flip: bool, flip_direction: str = 'horizontal') → Tensor

Map bboxes from testing scale to original image scale.

mmdet.structures.bbox.bbox_overlaps(bboxes1, bboxes2, mode='iou', is_aligned=False, eps=1e-06)

Calculate overlap between two set of bboxes.

FP16 Contributed by https://github.com/open-mmlab/mmdetection/pull/4889 .. note:

Assume bboxes1 is M x 4, bboxes2 is N x 4, when mode is 'iou',
there are some new generated variable when calculating IOU
using bbox_overlaps function:

1) is_aligned is False
    area1: M x 1
    area2: N x 1
    lt: M x N x 2
    rb: M x N x 2
    wh: M x N x 2
    overlap: M x N x 1
    union: M x N x 1
    ious: M x N x 1

    Total memory:
        S = (9 x N x M + N + M) * 4 Byte,

    When using FP16, we can reduce:
        R = (9 x N x M + N + M) * 4 / 2 Byte
        R large than (N + M) * 4 * 2 is always true when N and M >= 1.
        Obviously, N + M <= N * M < 3 * N * M, when N >=2 and M >=2,
                   N + 1 < 3 * N, when N or M is 1.

    Given M = 40 (ground truth), N = 400000 (three anchor boxes
    in per grid, FPN, R-CNNs),
        R = 275 MB (one times)

    A special case (dense detection), M = 512 (ground truth),
        R = 3516 MB = 3.43 GB

    When the batch size is B, reduce:
        B x R

    Therefore, CUDA memory runs out frequently.

    Experiments on GeForce RTX 2080Ti (11019 MiB):

    |   dtype   |   M   |   N   |   Use    |   Real   |   Ideal   |
    |:----:|:----:|:----:|:----:|:----:|:----:|
    |   FP32   |   512 | 400000 | 8020 MiB |   --   |   --   |
    |   FP16   |   512 | 400000 |   4504 MiB | 3516 MiB | 3516 MiB |
    |   FP32   |   40 | 400000 |   1540 MiB |   --   |   --   |
    |   FP16   |   40 | 400000 |   1264 MiB |   276MiB   | 275 MiB |

2) is_aligned is True
    area1: N x 1
    area2: N x 1
    lt: N x 2
    rb: N x 2
    wh: N x 2
    overlap: N x 1
    union: N x 1
    ious: N x 1

    Total memory:
        S = 11 x N * 4 Byte

    When using FP16, we can reduce:
        R = 11 x N * 4 / 2 Byte

So do the 'giou' (large than 'iou').

Time-wise, FP16 is generally faster than FP32.

When gpu_assign_thr is not -1, it takes more time on cpu
but not reduce memory.
There, we can reduce half the memory and keep the speed.

If is_aligned is False, then calculate the overlaps between each bbox of bboxes1 and bboxes2, otherwise the overlaps between each aligned pair of bboxes1 and bboxes2.

Parameters

• bboxes1 (Tensor) – shape (B, m, 4) in <x1, y1, x2, y2> format or empty.

• bboxes2 (Tensor) – shape (B, n, 4) in <x1, y1, x2, y2> format or empty. B indicates the batch dim, in shape (B1, B2, …, Bn). If is_aligned is True, then m and n must be equal.

• mode (str) – “iou” (intersection over union), “iof” (intersection over foreground) or “giou” (generalized intersection over union). Default “iou”.

• is_aligned (bool, optional) – If True, then m and n must be equal. Default False.

• eps (float, optional) – A value added to the denominator for numerical stability. Default 1e-6.

Returns

shape (m, n) if is_aligned is False else shape (m,)

Return type

Tensor

Example

>>> bboxes1 = torch.FloatTensor([
>>>     [0, 0, 10, 10],
>>>     [10, 10, 20, 20],
>>>     [32, 32, 38, 42],
>>> ])
>>> bboxes2 = torch.FloatTensor([
>>>     [0, 0, 10, 20],
>>>     [0, 10, 10, 19],
>>>     [10, 10, 20, 20],
>>> ])
>>> overlaps = bbox_overlaps(bboxes1, bboxes2)
>>> assert overlaps.shape == (3, 3)
>>> overlaps = bbox_overlaps(bboxes1, bboxes2, is_aligned=True)
>>> assert overlaps.shape == (3, )

Example

>>> empty = torch.empty(0, 4)
>>> nonempty = torch.FloatTensor([[0, 0, 10, 9]])
>>> assert tuple(bbox_overlaps(empty, nonempty).shape) == (0, 1)
>>> assert tuple(bbox_overlaps(nonempty, empty).shape) == (1, 0)
>>> assert tuple(bbox_overlaps(empty, empty).shape) == (0, 0)

mmdet.structures.bbox.bbox_project(bboxes: Union[Tensor, ndarray], homography_matrix: Union[Tensor, ndarray], img_shape: Optional[Tuple[int, int]] = None) → Union[Tensor, ndarray]

Geometric transformation for bbox.

Parameters

• bboxes (Union[torch.Tensor, np.ndarray]) – Shape (n, 4) for bboxes.

• homography_matrix (Union[torch.Tensor, np.ndarray]) – Shape (3, 3) for geometric transformation.

• img_shape (Tuple[int, int], optional) – Image shape. Defaults to None.

Returns

Converted bboxes.

Return type

Union[torch.Tensor, np.ndarray]

mmdet.structures.bbox.bbox_rescale(bboxes: Tensor, scale_factor: float = 1.0) → Tensor

Rescale bounding box w.r.t. scale_factor.

Parameters

• bboxes (Tensor) – Shape (n, 4) for bboxes or (n, 5) for rois

• scale_factor (float) – rescale factor

Returns

Rescaled bboxes.

Return type

Tensor

mmdet.structures.bbox.bbox_xyxy_to_cxcyah(bboxes: Tensor) → Tensor

Convert bbox coordinates from (x1, y1, x2, y2) to (cx, cy, ratio, h).

Parameters

bbox (Tensor) – Shape (n, 4) for bboxes.

Returns

Converted bboxes.

Return type

Tensor

mmdet.structures.bbox.bbox_xyxy_to_cxcywh(bbox: Tensor) → Tensor

Convert bbox coordinates from (x1, y1, x2, y2) to (cx, cy, w, h).

Parameters

bbox (Tensor) – Shape (n, 4) for bboxes.

Returns

Converted bboxes.

Return type

Tensor

mmdet.structures.bbox.cat_boxes(data_list: List[Union[Tensor, BaseBoxes]], dim: int = 0) → Union[Tensor, BaseBoxes]

Concatenate boxes with type of tensor or box type.

Parameters

data_list (List[Union[Tensor, BaseBoxes]]) –

A list of tensors or box types need to be concatenated. dim (int): The dimension over which the box are concatenated.

Defaults to 0.

Returns

obj`BaseBoxes`]: Concatenated results.

Return type

Union[Tensor,

mmdet.structures.bbox.convert_box_type(boxes: Union[ndarray, Tensor, BaseBoxes], *, src_type: Optional[Union[str, type]] = None, dst_type: Optional[Union[str, type]] = None) → Union[ndarray, Tensor, BaseBoxes]

Convert boxes from source type to destination type.

If boxes is a instance of BaseBoxes, the src_type will be set as the type of boxes.

Parameters

• boxes (np.ndarray or Tensor or BaseBoxes) – boxes need to convert.

• src_type (str or type, Optional) – source box type. Defaults to None.

• dst_type (str or type, Optional) – destination box type. Defaults to None.

Returns

Converted boxes. It’s type is consistent with the input’s type.

Return type

Union[np.ndarray, Tensor, BaseBoxes]

mmdet.structures.bbox.corner2bbox(corners: Tensor) → Tensor

Convert bbox coordinates from corners ((x1, y1), (x2, y1), (x1, y2), (x2, y2)) to (x1, y1, x2, y2).

Parameters

corners (Tensor) – Shape (n*4, 2) for corners.

Returns

Shape (n, 4) for bboxes.

Return type

Tensor

mmdet.structures.bbox.distance2bbox(points: Tensor, distance: Tensor, max_shape: Optional[Union[Sequence[int], Tensor, Sequence[Sequence[int]]]] = None) → Tensor

Decode distance prediction to bounding box.

Parameters

• points (Tensor) – Shape (B, N, 2) or (N, 2).

• distance (Tensor) – Distance from the given point to 4 boundaries (left, top, right, bottom). Shape (B, N, 4) or (N, 4)

• (Union[Sequence[int] (max_shape) – optional): Maximum bounds for boxes, specifies (H, W, C) or (H, W). If priors shape is (B, N, 4), then the max_shape should be a Sequence[Sequence[int]] and the length of max_shape should also be B.

• Tensor – optional): Maximum bounds for boxes, specifies (H, W, C) or (H, W). If priors shape is (B, N, 4), then the max_shape should be a Sequence[Sequence[int]] and the length of max_shape should also be B.

• Sequence[Sequence[int]]] – optional): Maximum bounds for boxes, specifies (H, W, C) or (H, W). If priors shape is (B, N, 4), then the max_shape should be a Sequence[Sequence[int]] and the length of max_shape should also be B.

:paramoptional): Maximum bounds for boxes, specifies

(H, W, C) or (H, W). If priors shape is (B, N, 4), then the max_shape should be a Sequence[Sequence[int]] and the length of max_shape should also be B.

Returns

Boxes with shape (N, 4) or (B, N, 4)

Return type

Tensor

mmdet.structures.bbox.empty_box_as(boxes: Union[Tensor, BaseBoxes]) → Union[Tensor, BaseBoxes]

Generate empty box according to input ``boxes` type and device.

Parameters

boxes (Tensor or BaseBoxes) – boxes with type of tensor or box type.

Returns

Generated empty box.

Return type

Union[Tensor, BaseBoxes]

mmdet.structures.bbox.find_inside_bboxes(bboxes: Tensor, img_h: int, img_w: int) → Tensor

Find bboxes as long as a part of bboxes is inside the image.

Parameters

• bboxes (Tensor) – Shape (N, 4).

• img_h (int) – Image height.

• img_w (int) – Image width.

Returns

Index of the remaining bboxes.

Return type

Tensor

mmdet.structures.bbox.get_box_tensor(boxes: Union[Tensor, BaseBoxes]) → Tensor

Get tensor data from box type boxes.

Parameters

boxes (Tensor or BaseBoxes) – boxes with type of tensor or box type. If its type is a tensor, the boxes will be directly returned. If its type is a box type, the boxes.tensor will be returned.

Returns

boxes tensor.

Return type

Tensor

mmdet.structures.bbox.get_box_type(box_type: Union[str, type]) → Tuple[str, type]

Get both box type name and class.

Parameters

box_type (str or type) – Single box type name or class.

Returns

A tuple of box type name and class.

Return type

Tuple[str, type]

mmdet.structures.bbox.get_box_wh(boxes: Union[Tensor, BaseBoxes]) → Tuple[Tensor, Tensor]

Get the width and height of boxes with type of tensor or box type.

Parameters

boxes (Tensor or BaseBoxes) – boxes with type of tensor or box type.

Returns

the width and height of boxes.

Return type

Tuple[Tensor, Tensor]

mmdet.structures.bbox.register_box(name: str, box_type: Optional[Type] = None, force: bool = False) → Union[Type, Callable]

Register a box type.

A record will be added to bbox_types, whose key is the box type name and value is the box type itself. Simultaneously, a reverse dictionary _box_type_to_name will be updated. It can be used as a decorator or a normal function.

Parameters

• name (str) – The name of box type.

• bbox_type (type, Optional) – Box type class to be registered. Defaults to None.

• force (bool) – Whether to override the existing box type with the same name. Defaults to False.

Examples

>>> from mmdet.structures.bbox import register_box
>>> from mmdet.structures.bbox import BaseBoxes

>>> # as a decorator
>>> @register_box('hbox')
>>> class HorizontalBoxes(BaseBoxes):
>>>     pass

>>> # as a normal function
>>> class RotatedBoxes(BaseBoxes):
>>>     pass
>>> register_box('rbox', RotatedBoxes)

mmdet.structures.bbox.register_box_converter(src_type: Union[str, type], dst_type: Union[str, type], converter: Optional[Callable] = None, force: bool = False) → Callable

Register a box converter.

A record will be added to box_converter, whose key is ‘{src_type_name}2{dst_type_name}’ and value is the convert function. It can be used as a decorator or a normal function.

Parameters

• src_type (str or type) – source box type name or class.

• dst_type (str or type) – destination box type name or class.

• converter (Callable) – Convert function. Defaults to None.

• force (bool) – Whether to override the existing box type with the same name. Defaults to False.

Examples

>>> from mmdet.structures.bbox import register_box_converter
>>> # as a decorator
>>> @register_box_converter('hbox', 'rbox')
>>> def converter_A(boxes):
>>>     pass

>>> # as a normal function
>>> def converter_B(boxes):
>>>     pass
>>> register_box_converter('rbox', 'hbox', converter_B)

mmdet.structures.bbox.roi2bbox(rois: Tensor) → List[Tensor]

Convert rois to bounding box format.

Parameters

rois (Tensor) – RoIs with the shape (n, 5) where the first column indicates batch id of each RoI.

Returns

Converted boxes of corresponding rois.

Return type

List[Tensor]

mmdet.structures.bbox.scale_boxes(boxes: Union[Tensor, BaseBoxes], scale_factor: Tuple[float, float]) → Union[Tensor, BaseBoxes]

Scale boxes with type of tensor or box type.

Parameters

• boxes (Tensor or BaseBoxes) – boxes need to be scaled. Its type can be a tensor or a box type.

• scale_factor (Tuple[float, float]) – factors for scaling boxes. The length should be 2.

Returns

Scaled boxes.

Return type

Union[Tensor, BaseBoxes]

mmdet.structures.bbox.stack_boxes(data_list: List[Union[Tensor, BaseBoxes]], dim: int = 0) → Union[Tensor, BaseBoxes]

Stack boxes with type of tensor or box type.

Parameters

data_list (List[Union[Tensor, BaseBoxes]]) –

A list of tensors or box types need to be stacked. dim (int): The dimension over which the box are stacked.

Defaults to 0.

Returns

obj`BaseBoxes`]: Stacked results.

Return type

Union[Tensor,

mask

class mmdet.structures.mask.BaseInstanceMasks

Base class for instance masks.

abstract property areas

areas of each instance.

Type

ndarray

abstract classmethod cat(masks: Sequence[T]) → T

Concatenate a sequence of masks into one single mask instance.

Parameters

masks (Sequence[T]) – A sequence of mask instances.

Returns

Concatenated mask instance.

Return type

T

abstract crop(bbox)

Crop each mask by the given bbox.

Parameters

bbox (ndarray) – Bbox in format [x1, y1, x2, y2], shape (4, ).

Returns

The cropped masks.

Return type

BaseInstanceMasks

abstract crop_and_resize(bboxes, out_shape, inds, device, interpolation='bilinear', binarize=True)

Crop and resize masks by the given bboxes.

This function is mainly used in mask targets computation. It firstly align mask to bboxes by assigned_inds, then crop mask by the assigned bbox and resize to the size of (mask_h, mask_w)

Parameters

• bboxes (Tensor) – Bboxes in format [x1, y1, x2, y2], shape (N, 4)

• out_shape (tuple[int]) – Target (h, w) of resized mask

• inds (ndarray) – Indexes to assign masks to each bbox, shape (N,) and values should be between [0, num_masks - 1].

• device (str) – Device of bboxes

• interpolation (str) – See mmcv.imresize

• binarize (bool) – if True fractional values are rounded to 0 or 1 after the resize operation. if False and unsupported an error will be raised. Defaults to True.

Returns

the cropped and resized masks.

Return type

BaseInstanceMasks

abstract expand(expanded_h, expanded_w, top, left)

See Expand.

abstract flip(flip_direction='horizontal')

Flip masks alone the given direction.

Parameters

flip_direction (str) – Either ‘horizontal’ or ‘vertical’.

Returns

The flipped masks.

Return type

BaseInstanceMasks

get_bboxes(dst_type='hbb')

Get the certain type boxes from masks.

Please refer to mmdet.structures.bbox.box_type for more details of the box type.

Parameters

dst_type – Destination box type.

Returns

Certain type boxes.

Return type

BaseBoxes

abstract pad(out_shape, pad_val)

Pad masks to the given size of (h, w).

Parameters

• out_shape (tuple[int]) – Target (h, w) of padded mask.

• pad_val (int) – The padded value.

Returns

The padded masks.

Return type

BaseInstanceMasks

abstract rescale(scale, interpolation='nearest')

Rescale masks as large as possible while keeping the aspect ratio. For details can refer to mmcv.imrescale.

Parameters

• scale (tuple[int]) – The maximum size (h, w) of rescaled mask.

• interpolation (str) – Same as mmcv.imrescale().

Returns

The rescaled masks.

Return type

BaseInstanceMasks

abstract resize(out_shape, interpolation='nearest')

Resize masks to the given out_shape.

Parameters

• out_shape – Target (h, w) of resized mask.

• interpolation (str) – See mmcv.imresize().

Returns

The resized masks.

Return type

BaseInstanceMasks

abstract rotate(out_shape, angle, center=None, scale=1.0, border_value=0)

Rotate the masks.

Parameters

• out_shape (tuple[int]) – Shape for output mask, format (h, w).

• angle (int | float) – Rotation angle in degrees. Positive values mean counter-clockwise rotation.

• center (tuple[float], optional) – Center point (w, h) of the rotation in source image. If not specified, the center of the image will be used.

• scale (int | float) – Isotropic scale factor.

• border_value (int | float) – Border value. Default 0 for masks.

Returns

Rotated masks.

shear(out_shape, magnitude, direction='horizontal', border_value=0, interpolation='bilinear')

Shear the masks.

Parameters

• out_shape (tuple[int]) – Shape for output mask, format (h, w).

• magnitude (int | float) – The magnitude used for shear.

• direction (str) – The shear direction, either “horizontal” or “vertical”.

• border_value (int | tuple[int]) – Value used in case of a constant border. Default 0.

• interpolation (str) – Same as in mmcv.imshear().

Returns

Sheared masks.

Return type

ndarray

abstract to_ndarray()

Convert masks to the format of ndarray.

Returns

Converted masks in the format of ndarray.

Return type

ndarray

abstract to_tensor(dtype, device)

Convert masks to the format of Tensor.

Parameters

• dtype (str) – Dtype of converted mask.

• device (torch.device) – Device of converted masks.

Returns

Converted masks in the format of Tensor.

Return type

Tensor

abstract translate(out_shape, offset, direction='horizontal', border_value=0, interpolation='bilinear')

Translate the masks.

Parameters

• out_shape (tuple[int]) – Shape for output mask, format (h, w).

• offset (int | float) – The offset for translate.

• direction (str) – The translate direction, either “horizontal” or “vertical”.

• border_value (int | float) – Border value. Default 0.

• interpolation (str) – Same as mmcv.imtranslate().

Returns

Translated masks.

class mmdet.structures.mask.BitmapMasks(masks, height, width)

This class represents masks in the form of bitmaps.

Parameters

• masks (ndarray) – ndarray of masks in shape (N, H, W), where N is the number of objects.

• height (int) – height of masks

• width (int) – width of masks

Example

>>> from mmdet.data_elements.mask.structures import *  # NOQA
>>> num_masks, H, W = 3, 32, 32
>>> rng = np.random.RandomState(0)
>>> masks = (rng.rand(num_masks, H, W) > 0.1).astype(np.int64)
>>> self = BitmapMasks(masks, height=H, width=W)

>>> # demo crop_and_resize
>>> num_boxes = 5
>>> bboxes = np.array([[0, 0, 30, 10.0]] * num_boxes)
>>> out_shape = (14, 14)
>>> inds = torch.randint(0, len(self), size=(num_boxes,))
>>> device = 'cpu'
>>> interpolation = 'bilinear'
>>> new = self.crop_and_resize(
...     bboxes, out_shape, inds, device, interpolation)
>>> assert len(new) == num_boxes
>>> assert new.height, new.width == out_shape

property areas

See BaseInstanceMasks.areas.

classmethod cat(masks: Sequence[T]) → T

Concatenate a sequence of masks into one single mask instance.

Parameters

masks (Sequence[BitmapMasks]) – A sequence of mask instances.

Returns

Concatenated mask instance.

Return type

BitmapMasks

crop(bbox)

See BaseInstanceMasks.crop().

crop_and_resize(bboxes, out_shape, inds, device='cpu', interpolation='bilinear', binarize=True)

See BaseInstanceMasks.crop_and_resize().

expand(expanded_h, expanded_w, top, left)

See BaseInstanceMasks.expand().

flip(flip_direction='horizontal')

See BaseInstanceMasks.flip().

pad(out_shape, pad_val=0)

See BaseInstanceMasks.pad().

classmethod random(num_masks=3, height=32, width=32, dtype=<class 'numpy.uint8'>, rng=None)

Generate random bitmap masks for demo / testing purposes.

Example

>>> from mmdet.data_elements.mask.structures import BitmapMasks
>>> self = BitmapMasks.random()
>>> print('self = {}'.format(self))
self = BitmapMasks(num_masks=3, height=32, width=32)

rescale(scale, interpolation='nearest')

See BaseInstanceMasks.rescale().

resize(out_shape, interpolation='nearest')

See BaseInstanceMasks.resize().

rotate(out_shape, angle, center=None, scale=1.0, border_value=0, interpolation='bilinear')

Rotate the BitmapMasks.

Parameters

• out_shape (tuple[int]) – Shape for output mask, format (h, w).

• angle (int | float) – Rotation angle in degrees. Positive values mean counter-clockwise rotation.

• center (tuple[float], optional) – Center point (w, h) of the rotation in source image. If not specified, the center of the image will be used.

• scale (int | float) – Isotropic scale factor.

• border_value (int | float) – Border value. Default 0 for masks.

• interpolation (str) – Same as in mmcv.imrotate().

Returns

Rotated BitmapMasks.

Return type

BitmapMasks

shear(out_shape, magnitude, direction='horizontal', border_value=0, interpolation='bilinear')

Shear the BitmapMasks.

Parameters

• out_shape (tuple[int]) – Shape for output mask, format (h, w).

• magnitude (int | float) – The magnitude used for shear.

• direction (str) – The shear direction, either “horizontal” or “vertical”.

• border_value (int | tuple[int]) – Value used in case of a constant border.

• interpolation (str) – Same as in mmcv.imshear().

Returns

The sheared masks.

Return type

BitmapMasks

to_ndarray()

See BaseInstanceMasks.to_ndarray().

to_tensor(dtype, device)

See BaseInstanceMasks.to_tensor().

translate(out_shape, offset, direction='horizontal', border_value=0, interpolation='bilinear')

Translate the BitmapMasks.

Parameters

• out_shape (tuple[int]) – Shape for output mask, format (h, w).

• offset (int | float, tuple) – The offset for translate.

• direction (str) – The translate direction, either “horizontal” or “vertical” or “both” in which case offset is tuple for (horizontal, vertical)

• border_value (int | float) – Border value. Default 0 for masks.

• interpolation (str) – Same as mmcv.imtranslate().

Returns

Translated BitmapMasks.

Return type

BitmapMasks

Example

>>> from mmdet.data_elements.mask.structures import BitmapMasks
>>> self = BitmapMasks.random(dtype=np.uint8)
>>> out_shape = (32, 32)
>>> offset = 4
>>> direction = 'horizontal'
>>> border_value = 0
>>> interpolation = 'bilinear'
>>> # Note, There seem to be issues when:
>>> # * the mask dtype is not supported by cv2.AffineWarp
>>> new = self.translate(out_shape, offset, direction,
>>>                      border_value, interpolation)
>>> assert len(new) == len(self)
>>> assert new.height, new.width == out_shape

class mmdet.structures.mask.PolygonMasks(masks, height, width)

This class represents masks in the form of polygons.

Polygons is a list of three levels. The first level of the list corresponds to objects, the second level to the polys that compose the object, the third level to the poly coordinates

Parameters

• masks (list[list[ndarray]]) – The first level of the list corresponds to objects, the second level to the polys that compose the object, the third level to the poly coordinates

• height (int) – height of masks

• width (int) – width of masks

Example

>>> from mmdet.data_elements.mask.structures import *  # NOQA
>>> masks = [
>>>     [ np.array([0, 0, 10, 0, 10, 10., 0, 10, 0, 0]) ]
>>> ]
>>> height, width = 16, 16
>>> self = PolygonMasks(masks, height, width)

>>> # demo translate
>>> new = self.translate((16, 16), 4., direction='horizontal')
>>> assert np.all(new.masks[0][0][1::2] == masks[0][0][1::2])
>>> assert np.all(new.masks[0][0][0::2] == masks[0][0][0::2] + 4)

>>> # demo crop_and_resize
>>> num_boxes = 3
>>> bboxes = np.array([[0, 0, 30, 10.0]] * num_boxes)
>>> out_shape = (16, 16)
>>> inds = torch.randint(0, len(self), size=(num_boxes,))
>>> device = 'cpu'
>>> interpolation = 'bilinear'
>>> new = self.crop_and_resize(
...     bboxes, out_shape, inds, device, interpolation)
>>> assert len(new) == num_boxes
>>> assert new.height, new.width == out_shape

property areas

Compute areas of masks.

This func is modified from detectron2. The function only works with Polygons using the shoelace formula.

Returns

areas of each instance

Return type

ndarray

classmethod cat(masks: Sequence[T]) → T

Concatenate a sequence of masks into one single mask instance.

Parameters

masks (Sequence[PolygonMasks]) – A sequence of mask instances.

Returns

Concatenated mask instance.

Return type

PolygonMasks

crop(bbox)

See BaseInstanceMasks.crop()

crop_and_resize(bboxes, out_shape, inds, device='cpu', interpolation='bilinear', binarize=True)

See BaseInstanceMasks.crop_and_resize()

expand(*args, **kwargs)

TODO: Add expand for polygon

flip(flip_direction='horizontal')

See BaseInstanceMasks.flip()

pad(out_shape, pad_val=0)

Padding has no effect on polygons`

classmethod random(num_masks=3, height=32, width=32, n_verts=5, dtype=<class 'numpy.float32'>, rng=None)

Generate random polygon masks for demo / testing purposes.

Adapted from [1]_

References

1

https://gitlab.kitware.com/computer-vision/kwimage/-/blob/928cae35ca8/kwimage/structs/polygon.py#L379 # noqa: E501

Example

>>> from mmdet.data_elements.mask.structures import PolygonMasks
>>> self = PolygonMasks.random()
>>> print('self = {}'.format(self))

rescale(scale, interpolation=None)

See BaseInstanceMasks.rescale()

resize(out_shape, interpolation=None)

See BaseInstanceMasks.resize()

rotate(out_shape, angle, center=None, scale=1.0, border_value=0, interpolation='bilinear')

See BaseInstanceMasks.rotate().

shear(out_shape, magnitude, direction='horizontal', border_value=0, interpolation='bilinear')

See BaseInstanceMasks.shear().

to_bitmap()

Convert polygon masks to bitmap masks.

to_ndarray()

Convert masks to the format of ndarray.

to_tensor(dtype, device)

See BaseInstanceMasks.to_tensor().

translate(out_shape, offset, direction='horizontal', border_value=None, interpolation=None)

Translate the PolygonMasks.

Example

>>> self = PolygonMasks.random(dtype=np.int64)
>>> out_shape = (self.height, self.width)
>>> new = self.translate(out_shape, 4., direction='horizontal')
>>> assert np.all(new.masks[0][0][1::2] == self.masks[0][0][1::2])
>>> assert np.all(new.masks[0][0][0::2] == self.masks[0][0][0::2] + 4)  # noqa: E501

mmdet.structures.mask.bitmap_to_polygon(bitmap)

Convert masks from the form of bitmaps to polygons.

Parameters

bitmap (ndarray) – masks in bitmap representation.

Returns

the converted mask in polygon representation. bool: whether the mask has holes.

Return type

list[ndarray]

mmdet.structures.mask.encode_mask_results(mask_results)

Encode bitmap mask to RLE code.

Parameters

mask_results (list) – bitmap mask results.

Returns

RLE encoded mask.

Return type

list | tuple

mmdet.structures.mask.mask2bbox(masks)

Obtain tight bounding boxes of binary masks.

Parameters

masks (Tensor) – Binary mask of shape (n, h, w).

Returns

Bboxe with shape (n, 4) of positive region in binary mask.

Return type

Tensor

mmdet.structures.mask.mask_target(pos_proposals_list, pos_assigned_gt_inds_list, gt_masks_list, cfg)

Compute mask target for positive proposals in multiple images.

Parameters

• pos_proposals_list (list[Tensor]) – Positive proposals in multiple images, each has shape (num_pos, 4).

• pos_assigned_gt_inds_list (list[Tensor]) – Assigned GT indices for each positive proposals, each has shape (num_pos,).

• gt_masks_list (list[BaseInstanceMasks]) – Ground truth masks of each image.

• cfg (dict) – Config dict that specifies the mask size.

Returns

Mask target of each image, has shape (num_pos, w, h).

Return type

Tensor

Example

>>> from mmengine.config import Config
>>> import mmdet
>>> from mmdet.data_elements.mask import BitmapMasks
>>> from mmdet.data_elements.mask.mask_target import *
>>> H, W = 17, 18
>>> cfg = Config({'mask_size': (13, 14)})
>>> rng = np.random.RandomState(0)
>>> # Positive proposals (tl_x, tl_y, br_x, br_y) for each image
>>> pos_proposals_list = [
>>>     torch.Tensor([
>>>         [ 7.2425,  5.5929, 13.9414, 14.9541],
>>>         [ 7.3241,  3.6170, 16.3850, 15.3102],
>>>     ]),
>>>     torch.Tensor([
>>>         [ 4.8448, 6.4010, 7.0314, 9.7681],
>>>         [ 5.9790, 2.6989, 7.4416, 4.8580],
>>>         [ 0.0000, 0.0000, 0.1398, 9.8232],
>>>     ]),
>>> ]
>>> # Corresponding class index for each proposal for each image
>>> pos_assigned_gt_inds_list = [
>>>     torch.LongTensor([7, 0]),
>>>     torch.LongTensor([5, 4, 1]),
>>> ]
>>> # Ground truth mask for each true object for each image
>>> gt_masks_list = [
>>>     BitmapMasks(rng.rand(8, H, W), height=H, width=W),
>>>     BitmapMasks(rng.rand(6, H, W), height=H, width=W),
>>> ]
>>> mask_targets = mask_target(
>>>     pos_proposals_list, pos_assigned_gt_inds_list,
>>>     gt_masks_list, cfg)
>>> assert mask_targets.shape == (5,) + cfg['mask_size']

mmdet.structures.mask.polygon_to_bitmap(polygons, height, width)

Convert masks from the form of polygons to bitmaps.

Parameters

• polygons (list[ndarray]) – masks in polygon representation

• height (int) – mask height

• width (int) – mask width

Returns

the converted masks in bitmap representation

Return type

ndarray

mmdet.structures.mask.split_combined_polys(polys, poly_lens, polys_per_mask)

Split the combined 1-D polys into masks.

A mask is represented as a list of polys, and a poly is represented as a 1-D array. In dataset, all masks are concatenated into a single 1-D tensor. Here we need to split the tensor into original representations.

Parameters

• polys (list) – a list (length = image num) of 1-D tensors

• poly_lens (list) – a list (length = image num) of poly length

• polys_per_mask (list) – a list (length = image num) of poly number of each mask

Returns

a list (length = image num) of list (length = mask num) of list (length = poly num) of numpy array.

Return type

list

mmdet.testing

mmdet.testing.demo_mm_inputs(batch_size=2, image_shapes=(3, 128, 128), num_items=None, num_classes=10, sem_seg_output_strides=1, with_mask=False, with_semantic=False, use_box_type=False, device='cpu', texts=None, custom_entities=False)

Create a superset of inputs needed to run test or train batches.

Parameters

• batch_size (int) – batch size. Defaults to 2.

• image_shapes (List[tuple], Optional) – image shape. Defaults to (3, 128, 128)

• num_items (None | List[int]) – specifies the number of boxes in each batch item. Default to None.

• num_classes (int) – number of different labels a box might have. Defaults to 10.

• with_mask (bool) – Whether to return mask annotation. Defaults to False.

• with_semantic (bool) – whether to return semantic. Defaults to False.

• device (str) – Destination device type. Defaults to cpu.

mmdet.testing.demo_mm_proposals(image_shapes, num_proposals, device='cpu')

Create a list of fake proposals.

Parameters

• image_shapes (list[tuple[int]]) – Batch image shapes.

• num_proposals (int) – The number of fake proposals.

mmdet.testing.demo_mm_sampling_results(proposals_list, batch_gt_instances, batch_gt_instances_ignore=None, assigner_cfg=None, sampler_cfg=None, feats=None)

Create sample results that can be passed to BBoxHead.get_targets.

mmdet.testing.demo_track_inputs(batch_size=1, num_frames=2, key_frames_inds=None, image_shapes=(3, 128, 128), num_items=None, num_classes=1, with_mask=False, with_semantic=False)

Create a superset of inputs needed to run test or train batches.

Parameters

• batch_size (int) – batch size. Default to 1.

• num_frames (int) – The number of frames.

• key_frames_inds (List) – The indices of key frames.

• image_shapes (List[tuple], Optional) – image shape. Default to (3, 128, 128)

• num_items (None | List[int]) – specifies the number of boxes in each batch item. Default to None.

• num_classes (int) – number of different labels a box might have. Default to 1.

• with_mask (bool) – Whether to return mask annotation. Defaults to False.

• with_semantic (bool) – whether to return semantic. Default to False.

mmdet.testing.get_detector_cfg(fname)

Grab configs necessary to create a detector.

These are deep copied to allow for safe modification of parameters without influencing other tests.

mmdet.testing.get_roi_head_cfg(fname)

Grab configs necessary to create a roi_head.

These are deep copied to allow for safe modification of parameters without influencing other tests.

mmdet.testing.random_boxes(num=1, scale=1, rng=None)

Simple version of kwimage.Boxes.random :returns: shape (n, 4) in x1, y1, x2, y2 format. :rtype: Tensor

References

https://gitlab.kitware.com/computer-vision/kwimage/blob/master/kwimage/structs/boxes.py#L1390 # noqa: E501

Example

>>> num = 3
>>> scale = 512
>>> rng = 0
>>> boxes = random_boxes(num, scale, rng)
>>> print(boxes)
tensor([[280.9925, 278.9802, 308.6148, 366.1769],
        [216.9113, 330.6978, 224.0446, 456.5878],
        [405.3632, 196.3221, 493.3953, 270.7942]])

mmdet.visualization

class mmdet.visualization.DetLocalVisualizer(name: str = 'visualizer', image: Optional[ndarray] = None, vis_backends: Optional[Dict] = None, save_dir: Optional[str] = None, bbox_color: Optional[Union[str, Tuple[int]]] = None, text_color: Optional[Union[str, Tuple[int]]] = (200, 200, 200), mask_color: Optional[Union[str, Tuple[int]]] = None, line_width: Union[int, float] = 3, alpha: float = 0.8)

MMDetection Local Visualizer.

Parameters

• name (str) – Name of the instance. Defaults to ‘visualizer’.

• image (np.ndarray, optional) – the origin image to draw. The format should be RGB. Defaults to None.

• vis_backends (list, optional) – Visual backend config list. Defaults to None.

• save_dir (str, optional) – Save file dir for all storage backends. If it is None, the backend storage will not save any data.

• bbox_color (str, tuple(int), optional) – Color of bbox lines. The tuple of color should be in BGR order. Defaults to None.

• text_color (str, tuple(int), optional) – Color of texts. The tuple of color should be in BGR order. Defaults to (200, 200, 200).

• mask_color (str, tuple(int), optional) – Color of masks. The tuple of color should be in BGR order. Defaults to None.

• line_width (int, float) – The linewidth of lines. Defaults to 3.

• alpha (int, float) – The transparency of bboxes or mask. Defaults to 0.8.

Examples

>>> import numpy as np
>>> import torch
>>> from mmengine.structures import InstanceData
>>> from mmdet.structures import DetDataSample
>>> from mmdet.visualization import DetLocalVisualizer

>>> det_local_visualizer = DetLocalVisualizer()
>>> image = np.random.randint(0, 256,
...                     size=(10, 12, 3)).astype('uint8')
>>> gt_instances = InstanceData()
>>> gt_instances.bboxes = torch.Tensor([[1, 2, 2, 5]])
>>> gt_instances.labels = torch.randint(0, 2, (1,))
>>> gt_det_data_sample = DetDataSample()
>>> gt_det_data_sample.gt_instances = gt_instances
>>> det_local_visualizer.add_datasample('image', image,
...                         gt_det_data_sample)
>>> det_local_visualizer.add_datasample(
...                       'image', image, gt_det_data_sample,
...                        out_file='out_file.jpg')
>>> det_local_visualizer.add_datasample(
...                        'image', image, gt_det_data_sample,
...                         show=True)
>>> pred_instances = InstanceData()
>>> pred_instances.bboxes = torch.Tensor([[2, 4, 4, 8]])
>>> pred_instances.labels = torch.randint(0, 2, (1,))
>>> pred_det_data_sample = DetDataSample()
>>> pred_det_data_sample.pred_instances = pred_instances
>>> det_local_visualizer.add_datasample('image', image,
...                         gt_det_data_sample,
...                         pred_det_data_sample)

add_datasample(name: str, image: ndarray, data_sample: Optional[DetDataSample] = None, draw_gt: bool = True, draw_pred: bool = True, show: bool = False, wait_time: float = 0, out_file: Optional[str] = None, pred_score_thr: float = 0.3, step: int = 0) → None

Draw datasample and save to all backends.

• If GT and prediction are plotted at the same time, they are

displayed in a stitched image where the left image is the ground truth and the right image is the prediction. - If show is True, all storage backends are ignored, and the images will be displayed in a local window. - If out_file is specified, the drawn image will be saved to out_file. t is usually used when the display is not available.

Parameters

• name (str) – The image identifier.

• image (np.ndarray) – The image to draw.

• data_sample (DetDataSample, optional) – A data sample that contain annotations and predictions. Defaults to None.

• draw_gt (bool) – Whether to draw GT DetDataSample. Default to True.

• draw_pred (bool) – Whether to draw Prediction DetDataSample. Defaults to True.

• show (bool) – Whether to display the drawn image. Default to False.

• wait_time (float) – The interval of show (s). Defaults to 0.

• out_file (str) – Path to output file. Defaults to None.

• pred_score_thr (float) – The threshold to visualize the bboxes and masks. Defaults to 0.3.

• step (int) – Global step value to record. Defaults to 0.

class mmdet.visualization.TrackLocalVisualizer(name: str = 'visualizer', image: Optional[ndarray] = None, vis_backends: Optional[Dict] = None, save_dir: Optional[str] = None, line_width: Union[int, float] = 3, alpha: float = 0.8)

Tracking Local Visualizer for the MOT, VIS tasks.

Parameters

• name (str) – Name of the instance. Defaults to ‘visualizer’.

• image (np.ndarray, optional) – the origin image to draw. The format should be RGB. Defaults to None.

• vis_backends (list, optional) – Visual backend config list. Defaults to None.

• save_dir (str, optional) – Save file dir for all storage backends. If it is None, the backend storage will not save any data.

• line_width (int, float) – The linewidth of lines. Defaults to 3.

• alpha (int, float) – The transparency of bboxes or mask. Defaults to 0.8.

add_datasample(name: str, image: ndarray, data_sample: DetDataSample = None, draw_gt: bool = True, draw_pred: bool = True, show: bool = False, wait_time: int = 0, out_file: Optional[str] = None, pred_score_thr: float = 0.3, step: int = 0) → None

Draw datasample and save to all backends.

• If GT and prediction are plotted at the same time, they are

displayed in a stitched image where the left image is the ground truth and the right image is the prediction. - If show is True, all storage backends are ignored, and the images will be displayed in a local window. - If out_file is specified, the drawn image will be saved to out_file. t is usually used when the display is not available. :param name: The image identifier. :type name: str :param image: The image to draw. :type image: np.ndarray :param data_sample: A data

sample that contain annotations and predictions. Defaults to None.

Parameters

• draw_gt (bool) – Whether to draw GT TrackDataSample. Default to True.

• draw_pred (bool) – Whether to draw Prediction TrackDataSample. Defaults to True.

• show (bool) – Whether to display the drawn image. Default to False.

• wait_time (int) – The interval of show (s). Defaults to 0.

• out_file (str) – Path to output file. Defaults to None.

• pred_score_thr (float) – The threshold to visualize the bboxes and masks. Defaults to 0.3.

• step (int) – Global step value to record. Defaults to 0.

mmdet.visualization.get_palette(palette: Union[List[tuple], str, tuple], num_classes: int) → List[Tuple[int]]

Get palette from various inputs.

Parameters

• palette (list[tuple] | str | tuple) – palette inputs.

• num_classes (int) – the number of classes.

Returns

A list of color tuples.

Return type

list[tuple[int]]

mmdet.visualization.jitter_color(color: tuple) → tuple

Randomly jitter the given color in order to better distinguish instances with the same class.

Parameters

color (tuple) – The RGB color tuple. Each value is between [0, 255].

Returns

The jittered color tuple.

Return type

tuple

mmdet.visualization.palette_val(palette: List[tuple]) → List[tuple]

Convert palette to matplotlib palette.

Parameters

palette (List[tuple]) – A list of color tuples.

Returns

A list of RGB matplotlib color tuples.

Return type

List[tuple[float]]

mmdet.utils

class mmdet.utils.AvoidOOM(to_cpu=True, test=False)

Try to convert inputs to FP16 and CPU if got a PyTorch’s CUDA Out of Memory error. It will do the following steps:

1. First retry after calling torch.cuda.empty_cache().

2. If that still fails, it will then retry by converting inputs

to FP16.

3. If that still fails trying to convert inputs to CPUs.

In this case, it expects the function to dispatch to CPU implementation.

Parameters

• to_cpu (bool) – Whether to convert outputs to CPU if get an OOM error. This will slow down the code significantly. Defaults to True.

• test (bool) – Skip _ignore_torch_cuda_oom operate that can use lightweight data in unit test, only used in test unit. Defaults to False.

Examples

>>> from mmdet.utils.memory import AvoidOOM
>>> AvoidCUDAOOM = AvoidOOM()
>>> output = AvoidOOM.retry_if_cuda_oom(
>>>     some_torch_function)(input1, input2)
>>> # To use as a decorator
>>> # from mmdet.utils import AvoidCUDAOOM
>>> @AvoidCUDAOOM.retry_if_cuda_oom
>>> def function(*args, **kwargs):
>>>     return None

```

Note

1. The output may be on CPU even if inputs are on GPU. Processing

   on CPU will slow down the code significantly.

2. When converting inputs to CPU, it will only look at each argument

   and check if it has .device and .to for conversion. Nested structures of tensors are not supported.

3. Since the function might be called more than once, it has to be

   stateless.

retry_if_cuda_oom(func)

Makes a function retry itself after encountering pytorch’s CUDA OOM error.

The implementation logic is referred to https://github.com/facebookresearch/detectron2/blob/main/detectron2/utils/memory.py

Parameters

func – a stateless callable that takes tensor-like objects as arguments.

Returns

a callable which retries func if OOM is encountered.

Return type

func

mmdet.utils.all_reduce_dict(py_dict, op='sum', group=None, to_float=True)

Apply all reduce function for python dict object.

The code is modified from https://github.com/Megvii- BaseDetection/YOLOX/blob/main/yolox/utils/allreduce_norm.py.

NOTE: make sure that py_dict in different ranks has the same keys and the values should be in the same shape. Currently only supports nccl backend.

Parameters

• py_dict (dict) – Dict to be applied all reduce op.

• op (str) – Operator, could be ‘sum’ or ‘mean’. Default: ‘sum’

• group (torch.distributed.group, optional) – Distributed group, Default: None.

• to_float (bool) – Whether to convert all values of dict to float. Default: True.

Returns

reduced python dict object.

Return type

OrderedDict

mmdet.utils.allreduce_grads(params, coalesce=True, bucket_size_mb=-1)

Allreduce gradients.

Parameters

• params (list[torch.Parameters]) – List of parameters of a model

• coalesce (bool, optional) – Whether allreduce parameters as a whole. Defaults to True.

• bucket_size_mb (int, optional) – Size of bucket, the unit is MB. Defaults to -1.

mmdet.utils.collect_env()

Collect the information of the running environments.

mmdet.utils.compat_cfg(cfg)

This function would modify some filed to keep the compatibility of config.

For example, it will move some args which will be deprecated to the correct fields.

mmdet.utils.find_latest_checkpoint(path, suffix='pth')

Find the latest checkpoint from the working directory.

Parameters

• path (str) – The path to find checkpoints.

• suffix (str) – File extension. Defaults to pth.

Returns

File path of the latest checkpoint.

Return type

latest_path(str | None)

References

1

https://github.com/microsoft/SoftTeacher /blob/main/ssod/utils/patch.py

mmdet.utils.get_caller_name()

Get name of caller method.

mmdet.utils.get_test_pipeline_cfg(cfg: Union[str, ConfigDict]) → ConfigDict

Get the test dataset pipeline from entire config.

Parameters

cfg (str or ConfigDict) – the entire config. Can be a config file or a ConfigDict.

Returns

the config of test dataset.

Return type

ConfigDict

mmdet.utils.imshow_mot_errors(*args, backend: str = 'cv2', **kwargs)

Show the wrong tracks on the input image.

Parameters

backend (str, optional) – Backend of visualization. Defaults to ‘cv2’.

mmdet.utils.log_img_scale(img_scale, shape_order='hw', skip_square=False)

Log image size.

Parameters

• img_scale (tuple) – Image size to be logged.

• shape_order (str, optional) – The order of image shape. ‘hw’ for (height, width) and ‘wh’ for (width, height). Defaults to ‘hw’.

• skip_square (bool, optional) – Whether to skip logging for square img_scale. Defaults to False.

Returns

Whether to have done logging.

Return type

bool

mmdet.utils.reduce_mean(tensor)

“Obtain the mean of tensor on different GPUs.

mmdet.utils.register_all_modules(init_default_scope: bool = True) → None

Register all modules in mmdet into the registries.

Parameters

init_default_scope (bool) – Whether initialize the mmdet default scope. When init_default_scope=True, the global default scope will be set to mmdet, and all registries will build modules from mmdet’s registry node. To understand more about the registry, please refer to https://github.com/vbti-development/onedl-mmengine/blob/main/docs/en/tutorials/registry.md Defaults to True.

mmdet.utils.replace_cfg_vals(ori_cfg)

Replace the string “${key}” with the corresponding value.

Replace the “${key}” with the value of ori_cfg.key in the config. And support replacing the chained ${key}. Such as, replace “${key0.key1}” with the value of cfg.key0.key1. Code is modified from `vars.py < https://github.com/microsoft/SoftTeacher/blob/main/ssod/utils/vars.py>`_ # noqa: E501

Parameters

ori_cfg (mmengine.config.Config) – The origin config with “${key}” generated from a file.

Returns

The config with “${key}” replaced by the corresponding value.

Return type

updated_cfg [mmengine.config.Config]

mmdet.utils.setup_cache_size_limit_of_dynamo()

Setup cache size limit of dynamo.

Note: Due to the dynamic shape of the loss calculation and post-processing parts in the object detection algorithm, these functions must be compiled every time they are run. Setting a large value for torch._dynamo.config.cache_size_limit may result in repeated compilation, which can slow down training and testing speed. Therefore, we need to set the default value of cache_size_limit smaller. An empirical value is 4.

mmdet.utils.setup_multi_processes(cfg)

Setup multi-processing environment variables.

mmdet.utils.split_batch(img, img_metas, kwargs)

Split data_batch by tags.

Code is modified from <https://github.com/microsoft/SoftTeacher/blob/main/ssod/utils/structure_utils.py> # noqa: E501

Parameters

• img (Tensor) – of shape (N, C, H, W) encoding input images. Typically these should be mean centered and std scaled.

• img_metas (list[dict]) – List of image info dict where each dict has: ‘img_shape’, ‘scale_factor’, ‘flip’, and may also contain ‘filename’, ‘ori_shape’, ‘pad_shape’, and ‘img_norm_cfg’. For details on the values of these keys, see mmdet.datasets.pipelines.Collect.

• kwargs (dict) – Specific to concrete implementation.

Returns

a dict that data_batch split by tags,

such as ‘sup’, ‘unsup_teacher’, and ‘unsup_student’.

Return type

data_groups (dict)

mmdet.utils.sync_random_seed(seed=None, device='cuda')

Make sure different ranks share the same seed.

All workers must call this function, otherwise it will deadlock. This method is generally used in DistributedSampler, because the seed should be identical across all processes in the distributed group.

In distributed sampling, different ranks should sample non-overlapped data in the dataset. Therefore, this function is used to make sure that each rank shuffles the data indices in the same order based on the same seed. Then different ranks could use different indices to select non-overlapped data from the same data list.

Parameters

• seed (int, Optional) – The seed. Default to None.

• device (str) – The device where the seed will be put on. Default to ‘cuda’.

Returns

Seed to be used.

Return type

int

mmdet.utils.update_data_root(cfg, logger=None)

Update data root according to env MMDET_DATASETS.

If set env MMDET_DATASETS, update cfg.data_root according to MMDET_DATASETS. Otherwise, using cfg.data_root as default.

Parameters

• cfg (Config) – The model config need to modify

• logger (logging.Logger | str | None) – the way to print msg