Cloud Academy Team

HDInsight – Azure’s Hadoop Big Data Service

HDInsight - Azure's Hadoop Big Data Service

How can Azure HDInsight solve your big data challenges?

Big data refers to large volumes of fast-moving data in any format that haven’t yet been handled by your traditional data processing system. In other words, it refers to data which have Volume, Variety and Velocity (commonly termed as V3 in Big Data circles). Data can come from just about anywhere: application logs, sensor data, archived images, videos, streaming data like twitter trends, weather forecast data, astronomical data, biological genomes, and almost anything generated by human or machine. Handling data on this scale is a relatively new problem. Azure’s HDInsight is, appropriately, a new tool that aims to address this problem.

New Challenges and New Solutions: Coping with Variety and Velocity

Whether human-readable or not, managing fast moving data generated at a massive scale – while maintaining data integrity – will require a different kind of processing mechanism than we would have been used for a traditional database or data-mining. Older solutions handled Volume well enough, but Variety and Velocity are relatively new problems. Since not everyone can afford super computers, brute force approaches probably aren’t going to be practical. This challenge inspired the development of Hadoop, which made it possible to process big data on industry-standard servers, while guaranteeing reliable and scalable parallel and distributed computing on smaller budgets.
Without diving too deeply into the technologies of big data management, we can’t go on without at least mentioning the biggest and most important players:

  • Hadoop (batch processing),
  • NoSQL (HBase, MongoDB, and Cassandra for distributed non-ACID databases),
  • Storm and Kafka (real time streaming data),
  • Spark (in-memory distributed processing), and
  • Pig scripts and Hive queries.

Besides the standard Apache Hadoop framework, there are many vendors who provide customized Hadoop distributions, like Cloudera, Hortonworks, MapR, and Greenplum. Many established cloud vendors provide some variety of Hadoop Platform as a Service (PaaS), like AWS’s Amazon Elastic Map Reduce (EMR), and Google Big Query.
The purpose of this post is to introduce you to Azure HDInsight, which is based on the Hortonworks Data Platform (HDP). The flexibility of Azure cloud and the innovation of Hortonworks can make Azure HDInsight a very interesting and productive space to process your big data.

Azure’s Big Data Solutions

Azure provides various big data processing services. The most popular of them is HDInsight, which is an on-demand Hadoop platform powered by Hortonworks Data Platform (HDP). Besides HDInsight (on which we’re going to focus our attention in this post) Azure also offers:

  • Data Lake Analytics
  • Data Factory
  • SQL Data Warehouse
  • Data Catalog

Azure HDInsight: A Comprehensive Managed Apache Hadoop, Spark, R, HBase, and Storm Cloud Service

As we mentioned, Azure provides a Hortonworks distribution of Hadoop in the cloud. “Hadoop distribution” is a broad term used to describe solutions that include some MapReduce and HDFS platform, in addition to a full stack featuring Spark, NoSQL, Pig, Sqoop, Ambari, and Zookeeper. Azure HDInsight provides all those technologies as part of its Big Data service, and also integrates with business intelligence (BI) tools like Excel, SQL Server Analysis Services, and SQL Server Reporting Services.
As a distribution, HDInsight comes in two flavors: Ubuntu and Windows Server 2012. Users can manage Linux-based HDInsight clusters using Apache Ambari and, for Windows users, Azure provides a cluster dashboard.
Cluster configuration for HDInsight is categorized into four different offerings:
1. Hadoop Cluster for query and batch processing with MapReduce and HDFS.
HDInsight - hadoop cluster

Hadoop clusters are made up of two types of nodes: Head Nodes that run Name Nodes and Job Trackers in a cluster (two nodes/cluster minimum); and Worker Nodes – also called Data Nodes – representing a Hadoop cluster’s true work horses. The minimum number of worker nodes is one, and you can scale worker nodes up or down according to need.

2. HBase for NoSQL-based workloads. NoSQL is a term defining a certain kind of data store and data processing engine that do not rely on traditional ACID properties but, instead, on CAP theorem.
HDInsight - hbase cluster
3. Storm for streaming data processing (remember the twitter data or sensor data use-case).
HDInsight - storm cluster
4. HDInsight Spark for in-memory parallel processing for big data analytics. Use cases for HDInsight Spark are Interactive data analysis and BI, Iterative Machine Learning, Streaming and real-time data analysis etc. In recent days, Apache Spark has taken over Hadoop based data analytics because of its capability to handle complex algorithms, faster in-memory processing and graph computing.
Azure Spark services diagram
You can also customize the cluster with scripts employing various technologies and languages. Spark, Solr, R, Giraph, and Hue can be used to install additional components.  HDInsight clusters also include the following (Apache) components:

  • Amabri for cluster provisioning, management, and monitoring.
  • Hive – and SQL query-like language that runs against data which, in turn, converts to MapReduce algorithms to process the data on HDFS.
  • Pig for data processing using user-created scripts.
  • Zookeeper for cluster co-ordination and management in a distributed environment.
  • Oozie for workflow management.
  • Mahout for machine learning.
  • Sqoop (SQL on Hadoop) for data import and export from SQL storage.
  • Tez – a successor of MapReduce that runs on YARN (Yet-Another-Resource-Negotiator) for complex and acyclic graph processing.
  • Phoenix – a layer over HBase to query and analyze data kept in SQL stores. Unlike Hive, Phoenix transforms queries into NoSQL API calls for processing and then converts them for MapReduce programming.
  • HCatalog provides a relational view of data in HDFS. It often used with Hive.
  • Avro – a data serialization format for the .NET environment.

The current version of HDInsight is 3.4, other software versions in the stack are up to these version numbers:

HDInsight software stack versions
  (HDInsight software stack)

Conclusion

Azure HDInsight offers all the best big data management features for the enterprise cloud, and has become one of the most talked about Hadoop Distributions in use. While users can quickly scale clusters up or down according to their needs, they will pay only for resources they actually use, and avoid the capital costs required to provision complex hardware configurations and the professionals needed to maintain them. HDInsight lets you crunch all kinds of data – whether structured or not – overcoming the burden of Hadoop cluster configuration.

If you’d like to learn more about Big Data and Hadoop, why not take our Analytics Fundamentals for AWS course? We’re currently building some great Azure content, but in the meantime this course provides a solid foundation in Big Data analytics for AWS. Check it out today!

Learn Analytics Fundamentals for AWS: EMR

 

Cloud Academy