FaceSigns: Semi-Fragile Neural Watermarks for Media Authentication and Countering Deepfakes

*Paarth Neekhara, *Shehzeen Hussain, Xinqiao Zhang, Ke Huang, Julian McAuley, Farinaz Koushanfar

University of California, San Diego
San Diego State University
* Equal contribution

[Paper] [Demo]

FaceSigns is a semi-fragile image watermarking framework for media authentication. Instead of identifying and detecting fake media using visual artifacts, we propose to proactively embed a semi-fragile watermark into a real image so that we can prove its authenticity when needed. Our watermarking framework is designed to be fragile to facial manipulations or tampering while being robust to benign image-processing operations such as image compression, scaling, saturation, contrast adjustments etc. This allows images shared over the internet to retain the verifiable watermark as long as face-swapping or any other Deepfake modification technique is not applied.


Methodology

FaceSigns encoder-decoder framework is trained to embed a bit string into the image pixels such that:

  1. 1) The encoded image looks visually similar to the original image.
  2. 2) The message is recoverable when the image undergoes benign transformatinos (simulated using differentiable image operations during training)
  3. 3) The message is not recoverable when facial tampering is applied.

We develop a differentiable procedure to simulate facial watermark tampering during training (details mentioned in the paper). We train the framework to optimize loss functions corresponding to each of the above objectives. Using only a limited set of benign and malicious transformations during training, we find that our framework is able to generalize to unseen benign and malicious transformations and reliably identify DeepFake manipulations.


Results


FaceSigns is able to recover messages with a high bit recovery accuracy when transformations such as instagram filters and JPEG-compression are applied thereby demonstrating robustness to benign transformations. We then apply malicious transformations such as face-swapping using different deep learning and computer graphics techniques. We find that the face-swapping techniques completely break the embedded signature. This is in contrast to prior works on robust image watermarking and steganography that can decode signatures from even facially manipulated images. This selective fragility of FaceSigns makes it suitable for reliably identifying facial manipulation to images signed using the FaceSigns encoder.

Fig: Examples of original and signed images using FaceSigns encoder along with the residual visualization.


Fig: Examples of benign transformations like photo filters and JPEG compression


Fig: Examples of malicious DeepFake/FaceSwapping transformations


Citing our Work

@article{facesigns2022,
  title={{FaceSigns: Semi-Fragile Neural Watermarks for Media Authentication and Countering Deepfakes}},
  author={Neekhara, Paarth and Hussain, Shehzeen and Zhang, Xinqiao and Huang, Ke and McAuley, Julian and Koushanfar, Farinaz},
  journal={arXiv:2204.01960},
  year={2022}
}