Fine-Tuning SOTA Object Detection Models on Real-World Datasets

In our previous blog post in this series, we discussed state-of-the-art models for object detection: the architectures, the theory, and what makes YOLO12, YOLO26, and RF-DETR tick. If you want the theoretical background on these models, start there. This post is the practical follow-up: how to actually use these models, how to fine-tune them on diverse, specialized datasets that look nothing like their training data, and how to evaluate the results – all within PyCharm. Why fine-tune at all? Every pretrained detector you download was trained on some distribution of images, almost always COCO, which is ~118k training images of everyday scenes containing 80 common object categories (people, cars, dogs, chairs, etc.). Real-world deployment data rarely looks like COCO. Things that object detection might actually be applied to, such as damaged industrial cables, bone fractures on X-rays, or densely stacked soda bottles on a shelf, are: Out of vocabulary: “Bone fracture” is not one of COCO’s 80 classes, so the model literally has no output category for it. Out of visual distribution: X-ray imagery, industrial close-ups, and heavily occluded shelf scenes differ drastically from consumer photos in texture, viewpoint, and object density. Deploying a detector on off-distribution data therefore requires fine-tuning. But before we break the models, let’s establish that we get similar results on our hardware to the ones reported by developers. The models For the purposes of this experiment, we’ll focus on three current SOTA object detection families and examine two sizes of each model: Family Variants Implementation YOLO12 yolov12n, yolov12m Original authors’ repo YOLO26 yolo26n, yolo26m Ultralytics PyPI package RF-DETR RFDETRNano, RFDETRBase Roboflow PyPI package Sanity check: Reproducing COCO val2017 baselines We’re going to be working with six pretrained checkpoints: Two different sizes of each of the three models. To check that these models are behaving as expected, we evaluated all of them on the full 5,000-image COCO validation dataset ( val2017) to verify the numbers reported in the previous post: Model Params (M) mAP50 mAP50-95 Latency (ms) YOLOv12-N2.550.55480.402123.9 YOLO26-N2.570.54980.3952 12.3 YOLOv12-M19.670.6953 0.525972.4 YOLO26-M21.900.69060.518113.9 RF-DETR Nano30.470.67500.4835 12.4 RF-DETR Base32.170.7210 0.532512.9 Three things stand out even before we leave COCO behind: Larger models (mostly) have better performance. RF-DETR Base leads (0.5325 mAP50-95), but the medium YOLOs get remarkably close (0.5259 / 0.5181) with ~10M fewer parameters. YOLO26’s NMS-free design pays off in throughput. YOLO26-N is the fastest model in the lineup (12.3 ms latency) at essentially the same mAP50-95 as YOLOv12-N, which, despite being the smallest model here, is considerably slower (23.9 ms latency). Attention is expensive. (If you want more detail on the model architectures and how they affect performance, see the previous blog post in this series.) RF-DETR Nano is not “nano” by parameter count (~30M – more than YOLO26-M), but it is well-optimized: With 12.4 ms latency, it is the second-fastest overall. Published papers report optimized inference latency: That is, they measure the model’s forward pass in isolation, stripped of the surrounding stages of the object detection pipeline. We deliberately skipped that aggressive optimization so our numbers reflect what you’d actually see when deploying these models. As a result, our latency figures don’t line up with the benchmarks in the models’ white papers. There are two main reasons for this: Hardware: We used different hardware from the NVIDIA T4 GPU that serves as the de facto standard in object detection benchmarking. Unoptimized computation graph: We ran the models in their native framework rather than converting them to TensorRT. TensorRT compiles a network into a hardware-specific engine, fusing layers, selecting the fastest kernels for the target GPU, and optionally running in reduced precision. That can cut latency substantially, but the resulting engine is tied to one GPU and requires an extra build step, so it doesn’t represent how these models perform out of the box. Accuracy is a different story: While our latencies diverge from the published ones, our mAP50-95 results fall within reasonable noise bounds of the reported figures. Now that we’ve seen what our pretrained models can do on COCO, the dataset they were trained on, let’s see what happens when they’re tested off distribution. The datasets For evaluation, we used RF100-VL, a large-scale collection of 100 multimodal datasets covering concepts deliberately chosen to be rare in object detection models’ pretraining data. These datasets contain exactly the off-distribution targets we care about. These targets also mirror common real-life applications for object detection, giving us a realistic test of these models’ capabilities out in the wild. We picked three datasets that stress test the models in different ways: Dataset Domain Why it’s hard Classes cable-damageTechnical/industrialFine-grained damage types on visually similar backgrounds break, thunderbolt bone-fractureMedical (X-ray)Entirely different imaging modality; subtle features angle, fracture, line, messed_up_angle soda-bottlesRetailHeavy occlusion, many near-identical instances per image coca-cola, fanta, sprite Tutorial: Fine-tuning all three models in PyCharm 🙂 Step 1: Setting up the project One of the first challenges we had to overcome in this project was that the three implementations do not share a compatible set of dependencies. In particular, the two different generations of YOLO require different versions of the ultralytics package. PyCharm offers a clean solution for this: one PyCharm project with three isolated uv environments – one per model family. We’ll run our computations on a remote GPU. Configuring a remote interpreter in PyCharm follows the same workflow as a local one: the same dialog and the same dropdown as in the local case. Note that remote interpreters require PyCharm Professional; Community Edition supports local environments only. Firstly, we need to instantiate our three uv virtual environments via: cd yolov12 && uv venv .venv --python 3.11 cd yolov26 && uv venv .venv --python 3.11 cd rf-detr && uv venv .venv --python 3.11 Once your uv virtual environments exist, register each one as an existing interpreter. Go to Settings | Python | Interpreter, click Add Interpreter → Add Local Interpreter, choose Environment as Select existing, and point the interpreter field at that environment’s bin/python. PyCharm doesn’t create anything here, it just picks up the environment uv already built. Repeat for each environment. From then on, switching is a matter of picking one from the Settings | Python | Interpreter dropdown, or from the interpreter widget in the bottom-right-hand status bar. You can find the full list of dependencies required for each model in their respective project repositories. You can either install all the projects’ dependencies in PyCharm’s built-in Terminal tool window or install individual packages using the Python Packages tool window (including selecting specific versions of packages). You can access both of these tool windows by clicking the relevant icons in the lower left-hand corner of the PyCharm toolbar. For a step-by-step guide on setting up the environments for all three models, see our GitHub implementation of this tutorial. Step 2: Getting the datasets To obtain the out-of-COCO-distribution datasets, we can install our datasets via the rf-detr virtual environment, since it has roboflow as one of its core dependencies. We then set the Roboflow API key as an environment variable so that it is available to the API when downloading the datasets. pip install roboflow export ROBOFLOW_API_KEY="your_key_here" # you can get API key here: https://docs.roboflow.com/reference/authentication/authentication/find-your-roboflow-api-key After setting everything up, now you can run the Python script below to get the three datasets we’re going to use in our tutorial: import os from roboflow import Roboflow api_key = os.environ.get("ROBOFLOW_API_KEY") if not api_key: raise RuntimeError("ROBOFLOW_API_KEY is not set") DATASETS = [ "bone-fracture-7fylg", "cable-damage", "soda-bottles", ] VERSION = 2 # RF100 projects are generally published at version 2 FORMAT = "yolov8" # or "coco", "voc", "yolov5" rf = Roboflow(api_key=api_key) workspace = rf.workspace("rf100") for slug in DATASETS: print(f"Downloading {slug} ...") try: project = workspace.project(slug) dataset = project.version(VERSION).download(FORMAT) print(f" -> {dataset.location}") except Exception as e: print(f" !! failed: {e}") This script connects to the Roboflow cloud service via its Python API client and downloads three specified RF100 datasets in YOLOv8 format. It loops through each dataset, reports where successful downloads are saved, and prints an error if any download fails. Step 3: Getting a zero-shot baseline by using pretrained models on custom data Before fine-tuning, we’re going to evaluate the COCO-pretrained checkpoints directly on our three datasets, to see whether the fine-tuning is actually necessary. The result was unambiguous: The models predicted essentially nothing. Zero-shot mAP50-95 on the test splits of our three datasets: Model cable-damage bone-fracture soda-bottles RF-DETR Nano0.00040.00000.0027 RF-DETR Base0.00050.00000.0004 YOLOv12-N0.00070.00000.0266 YOLO26-N0.00000.00000.0033 YOLOv12-M0.00000.00000.0160 YOLO26-M0.00000.00000.0012 This is to be expected; it’s not a bug! As the models are closed-vocabulary detectors, that is, they have a finite number of predefined target classes, they physically cannot output a class like fracture that isn’t in their 80-class COCO head. This is the punchline of this whole post: A model scoring 0.72 mAP50 on COCO scores 0.00 on bone fractures. Pretrained ≠ deployable, even when the model is state of the art. Basic machine learning principles still apply, even in the age of AI! Step 4: Fine-tuning All models were fine-tuned on a single A100 GPU for 10 epochs. We used standard Ultralytics/RF-DETR fine-tuning pipelines in order to fine-tune the models on our three datasets. We fine-tuned a model for each dataset. The full fine-tuning pipeline can be found in finetune_rf100.py scripts in the project repo, under the folders for each model. You can see the core of the training setup below. Both YOLO and RF-DETR are built on PyTorch under the hood, but the training loops are abstracted behind higher-level library APIs: Ultralytics’ YOLO.train() for the YOLO models, and RF-DETR’s own train() functionality. YOLO12 and YOLO26 train_model = YOLO(args.model) train_res = train_model.train( data=str(yaml_path), epochs=args.epochs, imgsz=args.imgsz, batch=args.batch, device=args.device, project=args.project, name=run_name, exist_ok=True, verbose=False, ) RF-DETR ModelClass().train( dataset_dir=str(coco_dir), output_dir=str(output_dir), epochs=args.epochs, batch_size=args.batch_size, grad_accum_steps=args.grad_accum, lr=args.lr, resolution=resolution, early_stopping=True, checkpoint_interval=1, ) Step 5: Results Fine-tuning transforms the picture. You can see the results on the test set after training: On the left, we have the pretrained models’ results for the COCO validation dataset. As we showed earlier, accuracy (mAP50-95) fell between 0.39 and 0.53, and all models except for YOLOv12-M showed low latency. The fine-tuned models on the right showed a similar range of accuracy for the cable-damage and soda-bottle detection tasks, only falling lower for the bone-fracture task. Moreover, the fine-tuned models were comparable in latency to the pretrained models for their intended tasks, and for YOLOv12-M, they were even faster. This suggests that, after fine-tuning to the target domain, the models achieve performance that’s broadly comparable to the pretrained performance on their original training domain. Let’s now have a closer look at the fine-tuned models’ performance, breaking it down by mAP50 and mAP50-95 for the three separate RF-100 datasets: Model cable-damage bone-fracture soda-bottles RF-DETR Nano0.9195 (0.4391)0.2317 (0.1136)0.9617 (0.6223) RF-DETR Base 0.9281 ( 0.4456) 0.4474 ( 0.1915)0.9688 (0.6332) YOLOv12-N0.9236 (0.4378)0.0911 (0.0532)0.9677 (0.6343) YOLO26-N0.8165 (0.3681)0.0193 (0.0064)0.9148 (0.5896) YOLOv12-M0.8266 (0.3649)0.1500 (0.0635) 0.9706 ( 0.6422) YOLO26-M0.8707 (0.3896)0.2194 (0.1038)0.9596 (0.6304) What the numbers say: The soda-bottles target is the easy win. Every model lands in the 0.91–0.97 mAP50 band. This is likely due to the fact that the domain (consumer products in photos) is visually close to existing classes in COCO, so only the vocabulary was new. Interestingly, the attention model family does great here, with YOLOv12-M taking the top spot (0.6422 mAP50-95). cable-damage: Detection is easy, but localization is hard. mAP50 reaches 0.93, but mAP50-95 tops out at 0.446. It appears that the models find the damage reliably, yet they struggle to box thin, elongated defects precisely. If your application needs tight boxes at high IoU, this gap would be a significant issue. bone-fracture remains genuinely hard. The best model (RF-DETR Base, 0.447 mAP50) is far from production-ready, and the performance spread across models is huge. The modality shift from photos to X-rays means the pretrained backbone features transfer poorly. The different image modality and small, sometimes almost indistinguishable bone fractures make the detection task way harder than the one employed on common objects identification. This is the dataset that would most benefit from domain-specific pretraining, more data, or longer fine-tuning. RF-DETR Base is the most consistent performer, winning on two out of three datasets and challenging seriously for the third. The DETR-style architecture seems to transfer more robustly to unfamiliar domains. Qualitative results To visually assess how these models perform, we can overlay the predicted bounding boxes on the images. Let’s look at the objects our models detected in six random images per class: We can see this confirms the accuracy values we saw above: The noisy images of soda bottles in fridges are labeled accurately, with tight bounding boxes for each object. The cable damage is identified less consistently, with some models failing to find the damage altogether, and others creating unnecessarily large bounding boxes. Finally, the images of broken bones contrast sharply with the other two, with less than half of the images having any break identified, and different models identifying different potential breakage points. Conclusions Pretrained object detectors are powerful, based on advancements in model architecture over the past five years, but as we’ve seen here, pretrained does not necessarily mean deployable. All six models performed well on COCO, yet when we applied those same checkpoints directly to our specialized datasets, their performance fell close to zero. However, fine-tuning completely changed that picture. After only 10 epochs of fine-tuning, all three model families were able to adapt well to both the cable-damage and soda-bottle datasets. As we noted, the soda-bottle task was particularly transferable, likely because it contained objects similar to those contained in COCO. cable-damage was also detected relatively reliably, although the larger gap between mAP50 and mAP50-95 showed that precisely locating these tiny defects was still challenging for all of the models. However, bone-fracture was a completely different story, likely because moving from the sort of natural images contained in COCO to X-rays is a much larger domain shift. While RF-DETR handled this jump best, even its performance shows the limits of fine-tuning, and there are times when you might need to consider more data, longer training, or even domain-specific pretraining. The broader takeaway is that there is no single “best” detector: It is dependent on the task. Model size, latency requirements, licensing restrictions, and most importantly, the similarity between the model’s pretraining data and your target domain all affect the outcome. It is important to refrain from unquestioningly trusting the numbers reported by model providers and explore the fit of a specific model for your own particular task. Get started with PyCharm today In this post, we’ve gone from validating pretrained YOLO12, YOLO26, and RF-DETR checkpoints on COCO to testing them zero-shot on specialized data, to fine-tuning them on three very different object detection tasks, and then finally, comparing the resulting accuracy and latency. Along the way, we’ve seen how PyCharm can help manage the practical side of a project like this, where multiple model families require different dependency sets and training environments. PyCharm helps you keep these workflows together in a single project while using isolated Python environments for each model family. Its interpreter management, built-in terminal, Python Packages tool window, and support for remote development make it easier to move between environments and run training on remote GPU hardware without having to manage each part of this workflow separately. If you’d like to try these experiments yourself, maybe look into fine-tuning these models for your own specific object detection use case! PyCharm is available to download and try. You can use the accompanying project code to reproduce our COCO baselines, download the RF100 datasets, fine-tune the models, and evaluate them using the held-out test splits. You can find the full code for this project on GitHub. And if you’d like to learn more about object detection, including the architectures behind the models we used in this post, check out the previous post in this series.