AWS WAF - How to protect your site from DDoS

Web attacks like SQL injection and Cross-Site Scripting can be devastating, resulting in massive data breaches, customer turnover, notification costs, lawsuits, and fines.

AWS WAF is a web application firewall that lets you monitor the HTTP and HTTPS requests that are forwarded to Amazon CloudFront and lets you control access to your content. Based on conditions that you specify, such as the IP addresses that requests originate from, or the values of query strings, CloudFront responds to requests either with the requested content or with an HTTP 403 status code (Forbidden). You can also configure CloudFront to return a custom error page when a request is blocked.

                    

Image sourced from here.                                        Image sourced from here.

The first illustration shows an infrastructure trying to respond to all requests, an approach that exhausts the web server’s resources. The second illustration shows a resilient infrastructure that uses AWS WAF, which blocks requests originating from blacklisted sources.

AWS WAF is a relatively new service only recently brought out of Beta. This service is tightly coupled to the CloudFront CDN service. The WAF service reviews traffic that is passing through the CDN and, based off defined rules, tells the CDN to either block or allow the traffic. To use this service, all site traffic must pass through a CloudFront CDN.

AWS WAF helps in preventing from a lot of attacks, but DDoS is the most common form of attack and also the most difficult to curb, let us start with what exactly is a DDoS attack.

DDoS Attacks

A Denial of Service (DoS) attack is an attack that can make your website or application unavailable to end users. To achieve this, attackers use a variety of techniques that consume network or other resources, disrupting access for legitimate end users. In its simplest form, a DoS attack against a target  is executed by a lone attacker from a single source, as shown below:

Diagram of a DOS attack

Image sourced from here.

In the case of a Distributed Denial of Service (DDoS) attack, an attacker uses multiple sources—which may be compromised or controlled by a group of collaborators—to orchestrate an attack against a target. As illustrated below, in a DDoS attack, each of the collaborators or compromised hosts participates in the attack, generating a flood of packets or requests to overwhelm the intended target.

Diagram of a DDOS attack

Image sourced from here.

DDoS attacks are most common at layers 3, 4, 6, and 7 of the Open Systems Interconnection (OSI) model, which is described in the above table. Layer 3 and 4 attacks correspond to the Network and Transport layers of the OSI model. This distinction is important because the attack types directed at these layers are different and so different techniques are used to build resiliency.

Architecture Implications

WAF can be implemented as a CloudFormation stack as illustrated in the image below:

Image sourced from here.

The CloudFormation stack works as follows:

  1. All web requests are passed through a CloudFront CDN network. All request data is persisted to log files that reside on S3 buckets.
  2. For every new request log file that is persisted, a Lambda function is triggered to analyse the log file data. The Lambda function reviews the traffic patterns and then updates the WAF block list based off defined rules.
  3. The WAF device blocks entry to the Elastic Load Balancer based off the source IP address.

Features

Web Traffic Filtering

Traffic filtering is accomplished by creating specific web request conditions, which are then grouped into rules. These rules are then associated with a CloudFront distribution through a web access control list.

Conditions

Conditions define the basic characteristics that you want AWS WAF to watch for in web requests:

  1. Scripts that are likely to be malicious. Attackers embed scripts that can exploit vulnerabilities in web applications; this is known as cross-site scripting.
  2. The IP addresses or address ranges that requests originate from.
  3. The length of specified parts of the request (Header, HTTP method, URI, body or query string).
  4. SQL code that is likely to be malicious. Attackers try to extract data from your database by embedding malicious SQL code in a web request; this is known as SQL injection.
  5. Strings that appear in the request(Header, HTTP method, URI, body or query string), for example, values that appear in the User-Agent header or text strings that appear in the query string.

Some conditions take multiple values. For example, you can specify up to 1000 IP addresses or IP address ranges in an IP condition.

Rules

You combine conditions into rules to precisely target the requests that you want to allow or block.

When a rule includes multiple conditions, AWS WAF looks for requests that match all those conditions — it ANDs the conditions together.

Web Access Control Lists (ACLs)

Finally, you combine rules into a Web ACL. This is where you define an action for each rule—allow, block, or count—and a default action. A Web ACL is also associated to a CloudFront resource. This allows you to have a set of rules and actions for multiple web sites.

When a web request matches all of the conditions in a rule, AWS WAF can either allow the request to be forwarded to CloudFront or block the request. For testing purposes, you can instruct WAF to count the requests and evaluate their behaviour later. You specify the action that you want AWS WAF to perform for each rule.

At the simplest level, AWS WAF lets you choose one of the following behaviours:

  1. Allow all requests except the ones that you specify – This is useful when you want CloudFront to serve content for a public website but you also want to block requests from attackers.
  2. Block all requests except the ones that you specify – This is useful when you want CloudFront to serve content for a restricted website whose users are readily identifiable by properties in web requests, such as the IP addresses they use to browse to the website.
  3. Count the requests that match the properties that you specify – When you want to allow or block requests based on new properties in web requests, you can first configure AWS WAF to count the requests that match those properties without allowing or blocking those requests. This lets you confirm that you didn't accidentally configure AWS WAF to block all of the traffic to your website. When you're confident that you specified the correct properties, you can change the behaviour to allow or block requests.

Full Feature API

Combining the core WAF Web Traffic Filtering features with some of the AWS services you can make the rules dynamic. For example, it is possible to temporarily block IP Addresses based off request volume - shutting down bots or screen scraping processes.

There are several CloudFormation templates that can jump-start setting up some of these dynamic rules.

You can view the AWF pricing list and limits on entities here and here.


Computer Vision - Face Detection

COMPUTER VISION

What is it...?

Computer Vision is mimicking the abilities of human vision by electronically perceiving and understanding an image.

It is a broad term and includes a lot of domains like Gesture Recognition, Optical Character Recognition, Face detection and a lot more.

In this article, we will be focussing on face detection and try to understand the key ideas that allow us to detect human faces in real time.


It all begins with a pixel!

As shown below, a digital representation of an image comprises a large number of pixels depending on the resolution of the image.

Each pixel represents the smallest unit containing the information about how the image will be rendered on a digital device.

Each pixel can be represented by 4 bytes ( 1 byte each for red, green, blue and alpha ).


What Face Detection is...?

It is essentially processing a raw image to :

  • Detect the presence of human faces, if any.
  • Extract info about the coordinates and size of those human faces.

How do we do it...?

That's what this article is all about.

Before we dive into the implementation details, let’s discuss the framework that most of the modern face detection systems use... the Viola-Jones Object Detection Framework.


VIOLA JONES OBJECT DETECTION FRAMEWORK

The training is slow, but the detection is fast.

This framework introduced three key ideas:

  1. Integral image representation
  2. Construction of classifier by Adaptive Boosting
  3. Combining successively more complex classifiers in a cascade structure

Let's see what each of these are...


INTEGRAL IMAGE REPRESENTATION

Features instead of Pixels

This framework focuses on using features rather than pixel for computing.

What is a feature...?

Let's see an example to understand what a feature is...

As explained here, all human faces share some similar properties:

  • The eye region is darker than the upper cheeks.

  • The nose bridge region is brighter than the eyes.

Using features has an obvious benefit in facilitating training data because they can encode critical domain knowledge.

The way these features are used involves computation of difference between the sum of intensities of the pixels in light and dark regions.

Value = Σ (pixels in black area) - Σ (pixels in white area)

The above step is a key operation as it is repeated a number of times with regions of varying sizes and coordinates. Therefore, it certainly needs to be efficient to achieve overall efficiency.

This is where the idea of Integral image representation comes handy.

What is an Integral Image representation...?

An intermediate representation which allows us to quickly compute intensities of an area independently of the size of the region with a computational complexity of O(1) instead of O(n)

As explained here

  • The value of the integral image at a point is the sum of all the pixels above and to the left.
  • The sum of the pixels within a rectangle can be computed with four array references.


CONSTRUCTION OF CLASSIFIER BY ADAPTIVE BOOSTING

What is a classifier...?

As explained here, classifier is a function that takes the values of various features in an example (image to be tested in our case) and predicts the class that that example belongs to (whether it contain human face or not, in our case).

A classifier is backed by some training data ( which is usually the output of some machine learning algorithm ) and its efficiency and accuracy depends on the quality of the training data.

  • A classifier is called a weak classifier if it cannot be used alone to predict the class to which an example belongs to. It is generally computationally economical.
  • On the contrary, a classifier is called a strong classifier if it can be used alone to classify the example. It is generally computation intensive.

There can be a large number of rectangle features associated with each image sub-window.... in fact, they can be far larger than the number of pixels.

As Viola and Jones hypothesized, a relatively smaller number of features could be used to form an effective classifier.

The big question - Which features to select?

Viola and Jones used variant of AdaBoost to select the features as well as to train the classifier

What is AdaBoost... ?

Adaptive Boosting a.k.a AdaBoost is a learning algorithm which is used to boost the performance of a simple classifier.

Below is a conceptual overview of how adaptive boosting works:

  • The training data ( a collection of positive and negative samples i.e. the images with and without a human face ) is fed into a weak classifier.
  • After the first round of learning, the weights are normalized for the examples ( training data images ) to emphasize on those which were incorrectly classified by the previous classifier.
  • This process is repeated until we get the required accuracy of the classifier.
  • The final result is a strong classifier which is a linear combination of a number of weighted weak classifiers followed by a threshold.

Thankfully, for our purposes (human face detection), we do not need to train our own classifier.

Instead, we will be using the classifier training data provided by an open source OpenCV library.


THE ATTENTIONAL CASCADE

The key insight is that smaller ( and therefore more efficient ) boosted classifiers can be constructed which reject many of the negative sub-windows while detecting almost all positive instances.

The idea is to use simpler classifiers to reject a majority of negative sub-windows thereby focussing the attention to only the promising regions of the image.

The end result is an overall efficiency due to the reduction in input sub-windows for the computationally expensive classifiers.


SEEING IT IN ACTION...

Here is a video that visualizes the detection process of OpenCV's face detector.

The algorithm uses the Viola-Jones method :

  • An integral image is calculated.
  • Some calculations are done on all the areas defined by the black and white rectangles to analyze the differences between the dark and light regions of a face.
  • The sub-window (in red) is scanned across the image at various scales to detect if there is a potential face within the window.
  • If not, it continues scanning.
  • If it passes all stages in the cascade file, it is marked with a red rectangle.
  • In the post-processing stage, all the potential faces are checked for overlaps.

HOW DO I USE FACE-DETECTION IN MY WEB PROJECT...?

We have survived the theory, let's come to implementation part.

Depending on the nature of the requirement, we can go with one of the below options:

I. IMPLEMENT YOUR OWN CLOUD

The most flexible and powerful way would be to manage your own cloud solution.

  • You get to control the training of the classifiers and every aspect of it.
  • You may use some open source libraries like OpenCV which has implemented more than 2500 machine learning algorithms implemented and has C++, C, Python, Java and MATLAB interfaces... which mean you can easily interface it with node server via node-opencv
  • However, this approach has its own maintenance overheads which become critical once you scale your app.
  • Another potential drawback would be that the image needs to be transmitted over the network to the server. In simple scenarios e.g. tagging of friends etc.. we might go with implementing a client side solution only.

II. USE A THIRD PARTY LIBRARY

  • You may easily get face detection up and running by delegating all the maintenance and setup chores to some third party service providers like Google Cloud Vison API or the one provided by Microsoft.
  • These providers already have an exhaustive training data backing up their classifiers. They also support advanced features such as Explicit Content Detection.
  • These services are usually paid.
  • This solution is highly recommended till the pay as you use amounts balances out the resources to set up and train your own classifier which usually takes some time to fine tune.
  • The common potential drawback of the image transmitted over network remains.

III. USE A CLIENT SIDE JS LIBRARY

  • The computation is delegated to the client but the image is not transmitted over the network to detect faces.
  • This solution is particularly valuable when we need to support simple requirements like tagging of friends, or focussing on an area of the image with the face etc.
  • One of the popular libraries to support this is trackingjs
  • Simplest to implement.

USING TRACKINGJS TO IMPLEMENT FACE DETECTION

You can install this library by simply typing

npm install tracking

Once installed, this library has the following files in the build directory

  • The files in the data folder contain the classifiers. You need to include the appropriate classifier based on what you need to detect.
  • So basically, we need to include the tracking library and one ( or more depending on use case ) of the classifiers and we are all set.

The classifier data looks like as shown below.

Below is a code snippet from the examples provided on the trackingjs site.

Using this library involves the following steps:

  • Require trackingjs and appropriate classifier ( from the data directory )
  • Instantiate a tracking.ObjectTracker with the name of the classifier to use.
  • Set appropriate callback on the tracker for the track event.
  • Trigger the tracker by simply invoking tracking.track with the element ( can be image / video / canvas ) and the tracker as the arguments.
    window.onload = function() {
      var img = document.getElementById('img');
      // We need to detect face... instantiate a new ObjectTracker with 'face' as an argument
      // IMPORTANT: We need to include appropriate classifier.
      var tracker = new tracking.ObjectTracker('face');

      // Set up appropriate listeners for track event
      tracker.on('track', function(event) {
        event.data.forEach(function(rect) {
          // Do something with the faces identified in the image.
          plotRectangle(rect.x, rect.y, rect.width, rect.height);
        });
      });

      // Invoke the track
      tracking.track(img, tracker);
    }

So that was a beginner's introduction to real-time face detection on the web.

Hope you found it interesting. Thanks for reading.


24/2 - 14 things I learnt from gmail Outage

Source : PC Disorder

Web version of Gmail is down (working with some IMAP accounts). And I am  learning few things here :

1. How to back up your gmail account ?

2. Developers and Server Administrators are HUMANS. Even google can have occasional server glitches.

3. There is something known as IMAP and POP.

4. Dont put your life - critical stuff on one web service even if it is Google. TRUST YOUR BACKUPS. Take Backups or store your data somewhere else. Whether its delicious or gmail or twitter. You dont know what you need in next moment. here is http://www.gmail-backup.com/

5. Life is more than just checking emails.

6. There is something known as gears Gmail offline with google Gears - http://lifehacker.com/5140668/gmail-goes-offline-with-google-gears
7. Its a rare outage. Just don't leave gmail back to your corporate emails again. Its one of the best SAAS in market.

8. We know gmail on iphone is working, but don't expect next iphone ad to feature this - "Gmail was working on iphone even when gmail was not working".

9. Acknoweldge fast if there is trouble : Google almost puts up an update on gmail help page as naturally people were looking at their help page.

"We're aware of a problem with Gmail affecting a number of users. This problem occurred at approximately 1.30AM Pacific Time. We're working hard to resolve this problem and will post updates as we have them. We apologize for any inconvenience that this has caused."

10. If  you have doubts that any of your web services is not working, don't disturb your co workers. visit twitter.com or Google apps status Dashboard

Updates:

11. Its the first anniversary of Amazon Web Services Outage. On 15 of February last year. February is not the good month for big companies.

12. Don't depend on any cloud computing service for customer services. Have a backup system ready  : "Today's Gmail outage is affecting our support system. Customers should login at  some_URL to contact us."

13. Use Gmail Paper , in case you are not using it.

14. If your gmail is not working, try gmail HTML https://mail.google.com/mail/h/




Calm down guys. Its good that email is down. Now do some productive work.


Web’s Horoscope for 2007

2007 Richard MacManus, Ebrahim Ezzy, Emre Sokullu, Alex Iskold and Rudy De Waele have made predictions of the trends to be followed in 2007 by web community. You will see Web 2.0 in its full blow in 2007. Wanna see the web2.0 speeding towards web 3.0 … jus follow the latest drift this year. Read all the predictions here. I am looking forward for the year of Google, Seach 2.0, RSS and of course last but not the least Rails 1.2 :)