Reverse image search

Reverse image search is a content-based image retrieval (CBIR) query technique that involves providing the CBIR system with a sample image that it will then base its search upon; in terms of information retrieval, the sample image is very useful in its ways. In particular, reverse image search is characterized by a lack of search terms. This effectively removes the need for a user to guess at keywords or terms that may or may not return a correct result. Reverse image search also allows users to discover content that is related to a specific sample image,[1] popularity of an image, and discover manipulated versions and derivative works.[2]

Reverse image search using Google Images.

Uses

Reverse image search may be used to:[3]

  • Locate the source of an image.
  • Find higher resolution versions.
  • Discover webpages where the image appears.
  • Find the content creator.
  • Get information about an image.

Algorithms

Commonly used reverse image search algorithms include:[4]

Yandex

Yandex Images offers a global reverse image and photo search. The site uses standard Content Based Image Retrieval (CBIR) technology used by many other sites, but additionally uses artificial intelligence-based technology to locate further results based on query.[6] Users can drag and drop images to the toolbar for the site to complete a search on the internet for similar looking images. The Yandex images searches some obscure social media sites in addition to more common ones offering content owners means of tracking plagiarism of image or photo intellectual property.

Google Images

Google's Search by Image is a feature that uses reverse image search and allows users to search for related images by uploading an image or copying the image URL. Google accomplishes this by analyzing the submitted picture and constructing a mathematical model of it. It is then compared with other images in Google's databases before returning matching and similar results. When available, Google also uses metadata about the image such as description. In 2022 the feature was replaced by Google Lens as the default visual search method on Google, and the old Search by Image function remains available within Google Lens.[7]

TinEye

TinEye is a search engine specialized for reverse image search. Upon submitting an image, TinEye creates a "unique and compact digital signature or fingerprint" of said image and matches it with other indexed images.[8] This procedure is able to match even very edited versions of the submitted image, but will not usually return similar images in the results.[9]

Pixsy

Pixsy reverse image search technology detects image matches[10] on the public internet for images uploaded to the Pixsy platform.[11] New matches are automatically detected and alerts sent to the user. For unauthorized use, Pixsy offers a compensation recovery service[12][13] for commercial use of the image owners work. Pixsy partners with over 25 law firms and attorneys around the world to bring resolution for copyright infringement. Pixsy is the strategic image monitoring service for the Flickr platform and users.[14]

eBay

eBay ShopBot uses reverse image search to find products by a user uploaded photo. eBay uses a ResNet-50 network for category recognition, image hashes are stored in Google Bigtable; Apache Spark jobs are operated by Google Cloud Dataproc for image hash extraction; and the image ranking service is deployed by Kubernetes.[15]

SK Planet

SK Planet uses reverse image search to find related fashion items on its e-commerce website. It developed the vision encoder network based on the TensorFlow inception-v3, with speed of convergence and generalization for production usage. A recurrent neural network is used for multi-class classification, and fashion-product region-of interest detection is based on Faster R-CNN. SK Planet's reverse image search system is built in less than 100 man-months.[16]

Alibaba

Alibaba released the Pailitao application during 2014. Pailitao (Chinese: 拍立淘, literally means shopping through a camera) allows users to search for items on Alibaba's E-commercial platform by taking a photo of the query object. The Pailitao application uses a deep CNN model with branches for joint detection and feature learning to discover the detection mask and exact discriminative feature without background disturbance. GoogLeNet V1 is employed as the base model for category prediction and feature learning.[17][18]

Pinterest

Pinterest acquired startup company VisualGraph in 2014 and introduced visual search on its platform.[19] In 2015, Pinterest published a paper at the ACM Conference on Knowledge Discovery and Data Mining conference and disclosed the architecture of the system. The pipeline uses Apache Hadoop, the open-source Caffe convolutional neural network framework, Cascading for batch processing, PinLater for messaging, and Apache HBase for storage. Image characteristics, including local features, deep features, salient color signatures and salient pixels are extracted from user uploads. The system is operated by Amazon EC2, and only requires a cluster of 5 GPU instances to handle daily image uploads onto Pinterest. By using reverse image search, Pinterest is able to extract visual features from fashion objects (e.g. shoes, dress, glasses, bag, watch, pants, shorts, bikini, earrings) and offer product recommendations that look similar.[20][21]

JD.com

JD.com disclosed the design and implementation of its real time visual search system at the Middleware '18 conference. The peer reviewed paper focuses on the algorithms used by JD's distributed hierarchical image feature extraction, indexing and retrieval system, which has 300 million daily active users. The system was able to sustain 80 million updates to its database per hour when it was deployed in production in 2018.[22]

Bing

Microsoft Bing published the architecture of their reverse image searching of system at the KDD'18 conference. The paper states that a variety of features from a query image submitted by a user are used to describe its content, including using deep neural network encoders, category recognition features, face recognition features, color features and duplicate detection features.[23]

Research systems

Microsoft Research Asia's Beijing Lab published a paper in the Proceedings of the IEEE on the Arista-SS (Similar Search) and the Arista-DS (Duplicate Search) systems. Arista-DS only performs duplicate search algorithms such as principal component analysis on global image features to lower computational and memory costs. Arista-DS is able to perform duplicate search on 2 billion images with 10 servers but with the trade-off of not detecting near duplicates.[24]

Open-source implementations

In 2007, the Puzzle library is released under the ISC license. Puzzle is designed to offer reverse image search visually similar images, even after the images have been resized, re-compressed, recolored and/or slightly modified.[25]

The image-match opensource project was released in 2016. The project, licensed under the Apache License, implements a reverse image search engine written in Python.[26]

Both the Puzzle library and the image-match projects use algorithms published at an IEEE ICIP conference.[27]

In 2019, a book published by O'Reilly documents how a simple reverse image search system can be built in a few hours. The book covers image feature extraction and similarity search, together with more advanced topics including scalability using GPUs and search accuracy improvement tuning.[28] The code for the system was made available freely on GitHub.[29]

The processing demands for performing reverse video search would be astoundingly high. There is no simple tool to just upload the video to find the matching results. At present there is no technoloy that can successfully perform a reverse video search.[30][31]

Production reverse image search systems
See also: List of CBIR engines

See also

References
  1. "How to search by image". Retrieved 2 November 2013.
  2. "Video searching with Frompo". Frompo.com. Retrieved 2 November 2013.
  3. "FAQ - TinEye - Why use TinEye?". TinEye.
  4. Bundling Features for Large Scale Partial-DuplicateWeb Image Search Microsoft.
  5. A New Web Image Searching Engine by Using SIFT Algorithm computer.org
  6. Raj, Abhishek, ed. (February 27, 2022). "How Does Yandex Reverse Image Search Work? Detailed Guide". www.buddinggeek.com. Budding Geek. Retrieved May 5, 2022.
  7. Li, Abner (10 August 2022). "Google Images on the web now uses Google Lens". 9to5Google. Retrieved 2 December 2022.
  8. "FAQ - TinEye - How does TinEye work?". TinEye.
  9. "FAQ - TinEye - Can TinEye find similar images??". TinEye.
  10. "Find stolen images - Pixsy". Pixsy. Retrieved 2017-10-20.
  11. "Pixsy.com review: Find & Fight Image Theft - Online Marketing for Artists -". Online Marketing for Artists. 2015-07-02. Retrieved 2017-10-20.
  12. "Pixsy: Find and Get Paid for Image Theft". artlawjournal.com. 2014-10-18. Retrieved 2017-10-20.
  13. "Resolve image theft - Pixsy". Pixsy. Retrieved 2017-10-20.
  14. "Flickr Teams Up with Pixsy to Get You Paid When Photos Are Stolen". petapixel.com. 9 April 2019. Retrieved 2019-12-12.
  15. Yang, Fan; Kale, Ajinkya; Bubnov, Yury; Stein, Leon; Wang, Qiaosong; Kiapour, Hadi; Piramuthu, Robinson (2017). Visual Search at eBay. acm.org. pp. 2101–2110. arXiv:1706.03154. doi:10.1145/3097983.3098162. ISBN 9781450348874. S2CID 22367645.
  16. Visual Fashion-Product Search at SK Planet
  17. Zhang, Yanhao; Pan, Pan; Zheng, Yun; Zhao, Kang; Zhang, Yingya; Ren, Xiaofeng; Jin, Rong (2018). Visual Search at Alibaba. acm.org. pp. 993–1001. arXiv:2102.04674. doi:10.1145/3219819.3219820. ISBN 9781450355520. S2CID 50776405.
  18. "Shopping With Your Camera: Visual Image Search Meets E-Commerce at Alibaba". Alibaba Tech. September 2020.
  19. Josh Constine (6 January 2014). "Pinterest Acquires Image Recognition And Visual Search Startup VisualGraph". TechCrunch. AOL.
  20. Jing, Yushi; Liu, David; Kislyuk, Dmitry; Zhai, Andrew; Xu, Jiajing; Donahue, Jeff; Tavel, Sarah (2015). Visual Search at Pinterest. acm.org. pp. 1889–1898. doi:10.1145/2783258.2788621. ISBN 9781450336642. S2CID 1153609.
  21. "Building a scalable machine vision pipeline". Pinterest Engineering. Archived from the original on 2015-09-06.
  22. Li, Jie; Liu, Haifeng; Gui, Chuanghua; Chen, Jianyu; Ni, Zhenyuan; Wang, Ning; Chen, Yuan (2018). The Design and Implementation of a Real Time Visual Search System on JD E-commerce Platform. acm.org. pp. 9–16. arXiv:1908.07389. doi:10.1145/3284028.3284030. ISBN 9781450360166. S2CID 53713854.
  23. Hu, Houdong; Wang, Yan; Yang, Linjun; Komlev, Pavel; Huang, Li; Chen, Xi (Stephen); Huang, Jiapei; Wu, Ye; Merchant, Meenaz; Sacheti, Arun (2018). Web-Scale Responsive Visual Search at Bing. acm.org. pp. 359–367. doi:10.1145/3219819.3219843. ISBN 9781450355520. S2CID 3427399.
  24. Duplicate-Search-Based Image Annotation Using Web-Scale Data Microsoft.
  25. The Puzzle library
  26. ProvenanceLabs / image-match
  27. An image signature for any kind of image
  28. Koul, Anirudh (October 2019). Practical Deep Learning for Cloud, Mobile, and Edge. O'Reilly Media. ISBN 9781492034865.
  29. Practical-Deep-Learning-Book source repository
  30. VHow to Use Reverse Video Search (& Why It’s Useful). searchenginejournal. September 2022.
  31. "How to Find Source of a Video with Reverse Image Search?". Alibaba DigitBin. October 2020.
  32. How to Do a Reverse Image Search From Your Phone
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.