For most cases, installation is as simple as:

pip3 install -U pip 
pip3 install -U fastdup

Latest fastdup is available for Mac M1 12 and higher, Ubuntu 20 and higher.

Legacy versions are: Windows, Ubutntu 18, Centos 07. Mac Intel.

Below we list common issues and prerequisites for various platforms:

Windows Server 2019 Native support
Ubuntu 20.04/18.04 Preliminaries
Mac OSX Preliminaries
CentOS 7 Preliminaries
Amazon Linux 2
Windows Server 10 Preliminaries
Pypi setup using pip
Preinstalled docker
Common installation errors

[New!] Added native Windows support for Windows Server 2019, Windows 10 and Windows 11 Pro

fastdup is currently supported on Ubuntu 20.04 or 18.04 OS, CentOS 7.9, Amazon Linux 2, RedHat 4.8, Mac OS 10+ using Intel or M1 CPUs, and Windows 10 Server (via WSL).

Windows Native support

Tested with Windows Server 2019, Windows 10 and Windows 11 Pro.

Currently we support windows server 2019 using Python3.8. Other windows are not fully tested. Let us know if you encountered issues using other versions
Assuming you like to work with minio cloud storage client, need to install it on windows and define environment variable FASTDUP_MC_PATH that points to the full path like C:\Users\my_username\mc.exe
For video extraction, need to install ffmpeg, and have environment variable FASTDUP_FFMPEG_PATH point to the full path of ffmpeg
For working with Amazon ec2, Need to verify you have bash.exe from Microsoft, it is found on default on C:\Windows\System32\bash.exe.
Then, install awscli via pip install awscli. Run bash.exe console, and inside the bash console run aws configure and put your ec2 credentials.
Warning: Unicode strings are partially supported. Please file any issue encountered when working with unicode strings.

Ubuntu 20.04/18.04 LTS Machine Setup

Required machine setup

sudo apt update
sudo apt -y install software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update
sudo apt -y install python3.8
sudo apt -y install python3-pip
sudo apt -y install libgl1-mesa-glx
python3.8 -m pip install --upgrade pip

Mac OS setup

# Install python:
brew install [email protected]
python3.9 -m pip install --U pip
python3.9 -m pip install fastdup

Note: when using conda on Mac M1 or Mac M1 Pro make sure you install conda for arm (and not conda for intel) otherwise it will not be possible to install.

Note: when using python installed via pyenv, python3.7 does not work since it is too old, use python3.8

CentOS 7 Setup / Amazon Linux 2 / RedHat 4.8

sudo yum -y install epel-release
sudo yum -y update
sudo yum -y groupinstall "Development Tools"
sudo yum -y install openssl-devel bzip2-devel libffi-devel xz-devel
sudo yum -y install wget
sudo yum install redhat-lsb-core # for lsb_release
sudo yum install -y ffmpeg ffmpeg-devel # for video support

Download and istall CentOS 7 whl image from our release page.

python3.7 -m pip install <path of the downloaded whl>

Amazon Linux 2 Setup

sudo yum install mesa-libGL -y

Download and isntall CentOS 7 whl image from our release page.

python3.7 -m pip install <path of the downloaded whl>

Windows 10 Server Setup Using WSL

Setting up WSL. The below instructions are for Windows Server 10. More detailed instructions are here.

For Windows 10+11 follow the instructions here.

Enable WSL on your machine (Search -> powershell-> right click -> run as administrator)

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform

Reboot your machine
Check that wsl is enabled using the command:

Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

Download ubuntu 18.04 from https://aka.ms/wslubuntu1804
After the download, run powershell as admin, goto your download folder (for example c:\users/danny_bickson/downloads/)

cd c:\\users\danny_bickson\Download # change to your download folders
Expand-Archive .\Ubuntu_1804.2019.522.0_x64.zip .\Ubuntu_1804
cd .\Ubuntu_1804\
 .\ubuntu1804.exe

This will take a few minutes, you will see an output of the kind:

Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: danny_bickson # (chnage to your username)
Enter new UNIX password: *******
Retype new UNIX password: *******
passwd: password updated successfully
Installation successful!
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Optional: Update WSL to version 2 (recommended, significant performance wins!)

Download the installer from https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi
Run the installer and follow the instructions

Once WSL and Ubuntu 18.04 are set up, continue with the below instructions

Inside the Ubuntu shell, run the following installers (you will be asked for password you entered before).

sudo apt update
sudo apt -y install software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update
sudo apt -y install python3.8
sudo apt -y install python3-pip
sudo apt -y install libgl1-mesa-glx
pip3 install --upgrade pip
python3.8 -m pip install fastdup

In case python fails to find fastdup, do the following:

Go to the latest version in pypi (for example https://pypi.org/project/fastdup/0.143/#files)
Download the file cp38-cp38-manylinux_2_27_x86_64.whl
python3.8 -m pip install fastdup-0.143-cp38-cp38-manylinux_2_27_x86_64.whl

Pip Package setup

Using pypi

This is the recommended installation method for all Mac, Ubuntu 18-20, Debian 10, Windows Server 10.
Will not work for Centos 7.9, RedHat 4.8.

python3.9 -m pip install -U pip
python3.9 -m pip install fastdup

Using stable release.

This is mandatory for CentOS 7.9 / RedHat 4.8 /Ubuntu 18 and similiar OS.

download the latest wheel for your system from our release page. Assuming the wheel file is found in your working folder, run:

python3.9 -m pip install -U pip
python3.9 -m pip install *.whl

Note: python used should be similar to your python. For example cp38 is python3.8, cp39 is python3.9 etc.

Debian package install

download the latest deb for your system from our release page. Assuming the wheel file is found in your working folder, run:

sudo dpkg -i fastdup-<VERSION>-ubuntu-20.04.deb

Application name is fastdup.

Docker

Pull from docker hub the latest ubuntu

docker pull karpadoni/fastdup-ubuntu-20.04

Build your own docker

docker build -f Dockerfile -t fastdup-ubuntu .

Currently supported software/hardware

Operating system

Ubuntu 20.04 LTS
Ubuntu 18.04 LTS
Mac OSX M1 Chip (tested on Big Sur)
Mac OSX M2 Chip (tested on Ventura)
Mac Intel Chip (tested on Mojave)
CentOS 7.9 (via stable release)
Amazon Linux 2 (via stable release)
Windows 10 Server (via WSL)

Software versions

Python 3.7, 3.8, 3.9 (via pip)
Note: conda is not recommended
pyenv installed python is not working on Mac, install python using brew install [email protected]

Hardware support

CPU (GPU not needed!)

Common installation issues and their solution

ERROR: fastdup-0.39-cp38-cp38-manylinux_2_31_x86_64.whl is not a supported wheel on this platform.

Check that you are on ubuntu 20.04 or 18.04 (via the command lsb_release -r). Alternatively on Mac M1 Big Sur or Mac Intel Mojave (use the command sw_vers)
Check that you are using the right python version (python3.8 and not python)
Make sure pip is up to date using python3.8 -m pip install -U pip).
Make sure you install using python3.8 -m pip install.. and not just pip install....
If that does not work, please open an issue with the otuput of python3.8 -m pip debug --verbose or join our slack channel.
If you are on mac using conda, make sure conda is installed for the right platform intel/ arm.
If you are on mac using pyenv, make sure to use python3.8 and not python3.7

ERROR on Ubuntu: libGL.so.1: cannot open shared object file: No such file or directory

Need to install depedency: sudo apt -y install libgl1-mesa-glx

Error on Mac+conda: OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/ zsh: abort python3

Solution from StackOverflow
You should install all packages without MKL support:

conda install nomkl
conda install numpy scipy pandas tensorflow
conda remove mkl mkl-service # may fail, don't worry

Error on Mac M1: AttributeError: partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline' (most likely due to a circular import)
Solution: downgrade your cv2 version to 4.5.5.64 using the command python.XX -m pip install -U opencv-python==4.5.5.64 where XX is your python version.

Error on Mac M1: library not loaded when trying to import cv2 ImportError: dlopen(/Users/mikasnoopy/homebrew/lib/python3.9/site-packages/cv2/python-3.9/cv2.cpython-39-darwin.so, 2): Library not loaded: /Users/mikasnoopy/homebrew/opt/dav1d/lib/libdav1d.5.dylib Referenced from: /Users/mikasnoopy/homebrew/opt/ffmpeg@4/lib/libavcodec.58.dylib Reason: image not found or any similar error.
Solution: Downgrade your ffmpg using brew remove ffmpeg; brew install ffmpeg@4

GitHub • Join Slack Community • Discussion Forum Blog • Documentation • About Us LinkedIn • Twitter