Home // International Journal On Advances in Security, volume 17, numbers 1 and 2, 2024 // View article

Pixels versus Privacy: Leveraging Vision-Language Models for Sensitive Information Extraction

Authors:
Sergej Schultenkämper
Frederik Simon Bäumer

Keywords: Computer Vision; Privacy; Social Networks

Abstract:
Threats to user privacy in Web 2.0 are abundant and can arise from various sources, including texts, geoinformation, images, videos, or combinations of these. To alert users of potential threats, it is crucial to gather all relevant information. However, aggregating user-specific information from various web platforms, including social networks, can be challenging due to the vast amount of data available, as well as issues with data quality and the numerous possible variants. This paper examines the capability of current Vision-Language Models to accurately identify relevant image data and extract sensitive information. To accomplish this, we developed our own dataset with diverse expressions for privacy attributes, based on the VISPR dataset. Furthermore, we address the challenge of synthetic images of people and its impact on our approach. Our findings suggest that these models are effective in pre-selecting relevant images, but there are limitations in information extraction.

Pages: 1 to 10

Copyright: Copyright (c) to authors, 2024. Used with permission.

Publication date: June 30, 2024

Published in: journal

ISSN: 1942-2636