Discussions
Fastdup version check: warning vs. RuntimeError
The RuntimeError returned by the version check in init.py forces users to upgrade to the newest versions of Fastdup, which may not always be compatible with other versions of libraries that the user has intentionally selected for other reasons. Because you release new versions of Fastdup so frequently, this error is encountered again very quickly even after the user resolves his or her dependency issues.
Ignore directory name patterns.
I'm running fastdup inside my Synology DS220+ NAS.
How to define the repeated images or how to select threshold?
Thank you for your project. I recently worked on a batch of data extracted from video captured by some fixed cameras(For object detection tasks, random date and random time sampling, with large intervals). Visualization reveals that this batch of data has a highly consistent background, with only a few different goals (target people, motor vehicles, non-motor vehicles). In fastdup, the result is usually above 0.9 points. The background will have an impact on the model, which may cause the model to learn some special background distribution information, and the generalization will be reduced due to the speculation (sorry I haven't had time to do the comparison experiment，and it may be tested within a week). Have you ever done a similar experiment? Perhaps the definition of duplicate images needs to be extended a bit for object detection tasks? I also used imagedups for a comparative test. (https://github.com/chinalu/imagedups use default parameter) .Of the 20,000 images, 515 were determined to be duplicated and cleared. The top 515 in fastdup scored 0.95 or more.How should I choose the threshold (in other experiments such as ocr/ license plate detection data, I found that 0.95 is not a good threshold)?
Welcome to fastdup V1.0!
Welcome, we are Visual-Layer, creators of Fastdup. We'd be happy to answer any questions, support topics and feature requests. We are also highly available on the Fastdup Slack channel