Advanced Facial Attributes Detection

Cloudinary provides an add-on for advanced face attribute detection capabilities, fully integrated into Cloudinary's image management and transformation pipeline. With the Advanced Facial Attribute Detection add-on, you can extend the Cloudinary features that involve semantic photo data extraction, image cropping and the positioning of image overlays. When using the Advanced Facial Attribute Detection add-on, your images are further processed and many advanced face attributes are automatically extracted. Cloudinary can then use these additional details to smartly crop, position, rotate and overlay images.

Cloudinary is a cloud-based service that provides an end-to-end image management solution including uploads, storage, transformations, optimizations and delivery, and offers a rich set of image management, transformation, analysis and cropping capabilities.

The Advanced Facial Attribute Detection add-on is an integrated face detection solution that utilizes Microsoft Cognitive Services. Microsoft Cognitive Services provides high precision face location detection with state-of-the-art cloud-based algorithms that can detect up to 64 human faces in an image. The detected faces are returned with rectangles (left, top, width and height) indicating the location of faces in the image in pixels, the exact position details of the eyes, mouth, eyebrows, nose and lips, as well as a series of face related attributes from each face such as pose, gender and age.

Getting started

Before you can use the Advanced Facial Attributes Detection add-on:

• You must have a Cloudinary account. If you don't already have one, you can sign up for a free account.

• Register for the add-on: make sure you're logged in to your account and then go to the Add-ons page. For more information about add-on registrations, see Registering for add-ons.

• Keep in mind that many of the examples on this page use our SDKs. For SDK installation and configuration details, see the relevant SDK guide.

• If you're new to Cloudinary, you may want to take a look at the Developer Kickstart for a hands-on, step-by-step introduction to Programmable Media features.

Face attribute detection on upload

The Advanced Facial Attribute Detection add-on is very simple to use. The extensive list of face attributes can be extracted from any photo when calling Cloudinary's upload API and setting the detection parameter to adv_face. For example, take a look at the following photo that we want to upload to Cloudinary's demo product environment:

When calling Cloudinary's API to upload the photo, the detection parameter is added to extract the face attributes in the uploaded photo:

The JSON snippet below contains an excerpt of the results of applying automatic face attribute detection on the uploaded image. The response includes very detailed information regarding each of the faces that were automatically detected in the photo:

• bounding_box - The bounding box surrounding a detected face.
• smile - The likelihood that the person is smiling.
• head_pose - The way the face is positioned in 3D.
• gender - Whether the person is a male or a female.
• age - Estimated age of the person.
• facial_hair - The likelihood that the person has a moustache, beard, and/or sideburns, respectively.
• glasses - Whether the person is wearing glasses.
• emotion - The likelihood that the person's expression implies certain emotions.
• blur - The amount of blur on the face.
• exposure - The photo exposure on the face.
• noise - A general and numeric evaluation of the amount of photographic noise on the face.
• makeup - Whether the person is wearing eye and/or lip makeup.
• accessories - An array of accessory items (if any) detected around the face area, such as headwear, glasses, or mask.
• occlusion - Whether the forehead, eyes, or mouth is blocked, for example by a scarf, hat, or mask.
• hair - Information about the hair, including whether the hair is visible in the photo, the likelihood that the person is bald, and the likelihood that the person's hair is a particular color.
• facial_landmarks - Exact position details of specific elements of the mouth, eyebrows, eyes, and nose.

Face attribute detection after uploading

The example above shows how to automatically detect face attributes of photos during their upload process. Alternatively, you can use Cloudinary's Admin API to apply automatic face attribute detection to already uploaded images, based on their public IDs. The following code sample uses Cloudinary's update API to apply face attribute detection to the uploaded image that has the 'coupled' public ID.

The face attributes that were previously extracted either using the 'upload' or 'update' API, are also available using the Admin API's show resource details method:

Face detection based cropping

Cloudinary provides a large set of image transformation and cropping options. Based on the position of facial attributes detected by the Advanced Facial Attribute Detection add-on, Cloudinary can crop your images to focus on the detected faces, while providing a large set of image transformation and cropping options when using a Cloudinary delivery URL or calling Cloudinary's image API.

To focus an automatic crop on the detected faces, simply set the crop parameter to thumb, fill or crop and the gravity parameter to adv_faces (set gravity to adv_face for focusing on the single largest detected face in the image).

The following code sample creates a 150x150 thumbnail of the coupled image that is generated using face-detection based cropping. The gravity parameter is set to adv_faces while the crop mode is set to thumb.

Alternatively, you can crop an image based on the single most visible face in a photo. The following code sample generates and delivers a 100x100 thumbnail containing a single face. The gravity parameter is set to adv_face while the crop mode is set to thumb.

Face detection with signed URLs

Cloudinary's dynamic image transformation URLs are quite powerful, however, due to the potential costs of users accessing unplanned dynamic URLs with the Advanced Facial Attribute Detection cropping directives, image transformation add-on URLs are required (by default) to be signed using Cloudinary's authenticated API and with the sign_url parameter set to true.

For example, a signed URL and the code to generate it:

As you can see, the generated Cloudinary URL includes a signature component (s--vnhlc4WH--). Only URLs with a valid signature (specific to that transformation) will be approved for on-the-fly image transformation using advanced facial attribute detection.

Eyes detection based cropping

In addition to face-detection based cropping, Cloudinary can dynamically crop your images based on the position of detected eyes. Simply set the gravity parameter to adv_eyes (g_adv_eyes for URLs) to center the image on the detected eyes. The example below delivers a 100x40 thumbnail centered on the eyes of the woman in the photo.

Eyes detection for accurate red eye removal

Cloudinary's rich transformation capabilities already allow you to automate red eye removal by setting the effect parameter to redeye (e_redeye for URLs). This enables smart red eye removal algorithms to be automatically applied on the fly to uploaded images.

In order to get even better quality results, you can use Cloudinary's Advanced Facial Attribute Detection add-on for eye detection together with the red eye removal effect. The add-on can automatically detect where eyes are located in a photo and then the red eye removal algorithm can be applied in a more precise way: simply set the effect parameter to adv_redeye (e_adv_redeye for URLs).

For example, an image named redeye was uploaded to Cloudinary. A cropped 300x80 thumbnail of the photo centered on the eyes only for illustration is displayed below:

Adding the redeye effect parameter and setting its value to adv_redeye delivers the following image:

Face overlays

The extracted facial attribute details also include the pose of the detected face, which allows Cloudinary to not only position overlays on top of detected faces, but also to rotate and scale the overlays accordingly for exact positioning over the underlying face. This is easily accomplished by setting the gravity parameter of the added overlay to adv_faces.

For example, the following PNG image named silver_face_mask was uploaded to Cloudinary:

To smartly overlay the mask image on top of all detected faces in the coupled image, the overlay parameter is set to the ID of the mask image and the gravity parameter is set to adv_faces. In addition, we set the region_relative flag together with a 1.1 width, which means that each overlay will be scaled to 110% of the width of the detected face.

Eyes overlays

As described above, the Advanced Facial Attribute Detection add-on detects specific facial attributes, including the exact position of the eyes of each face in a photo. Based on this information, Cloudinary can position overlays on top of all the detected eye pairs in an image.

For example, the following PNG image named glasses was uploaded to Cloudinary:

In order to smartly overlay the glasses on top of all detected eye pairs in an uploaded image, the overlay parameter is set to the ID of the glasses image and the gravity parameter is set to adv_eyes. In addition, we set the region_relative flag together with a 1.5 width, which means that each overlay should be scaled to 150% of the detected eyes.

In another example, the following PNG image named harlequinmask was uploaded to Cloudinary:

To smartly overlay the mask on top of all the detected eye pairs in the uploaded image cloudinary_team, the overlay parameter is set to the ID of the harlequinmask image and the gravity parameter is set to adv_eyes. We also set the region_relative flag together with a 1.7 width to scale the overlay to 170% of the width of the detected eyes, and resize the image to an oval thumbnail with a width of 700 pixels.

For a full list of additional Cloudinary's image transformation options, see Image transformations.