Skip Navigation Links

Course Length:
5 Days
Course Description:
You will learn how to use Apache Hadoop and write MapReduce programs. You will begin with a quick overview of installing Hadoop, setting it up in a cluster, and then proceed to writing data analytic programs. The course will present the basic concepts of MapReduce applications developed using Hadoop, including a close look at framework components, use of Hadoop for a variety of data analysis tasks, and numerous examples of Hadoop in action. The course will further examine related technologies such as Hive, Pig, and Apache Accumulo. Apache Accumulo is a highly scalable structured store based on Google's BigTable, written in Java and operates over the Hadoop Distributed File System (HDFS). Hive is data warehouse software for querying and managing large datasets. Pig is a platform to take advantage of parallelization when running data analysis. Finally, you will observe how Hadoop works in and supports cloud computing and explore examples with Amazon Web Services and case studies.
Who Should Attend:
The course is intended for programmers, architects, and project managers who have to process large amounts of data offline.
Benefits of Attendance:
Upon completion of this course, students will be able to:
  • Understand basic concepts of MapReduce applications developed using Hadoop
  • Understand how Hadoop works in and supports cloud computing and explore examples with Amazon Web Services and case studies
  • Use Apache Hadoop and write MapReduce programs
Prerequisites:
Students should have a basic familiarity with Linux administration and Java, as most code examples will be written in Java. Familiarity with basic statistical concepts (e.g. histogram, correlation) is helpful to appreciate the more advanced data processing examples.
Course Outline:
  • What is Hadoop?
    1. Understanding distributed systems and Hadoop
    2. Comparing SQL databases and Hadoop
    3. Understanding MapReduce
    4. Counting words with Hadoop—running your first program
    5. History of Hadoop
  • Starting Hadoop
    1. The building blocks of Hadoop
    2. Setting up SSH for a Hadoop cluster
    3. Running Hadoop
    4. Web-based cluster UI
  • Components of Hadoop
    1. Working with files in HDFS
    2. Anatomy of a MapReduce program
    3. Reading and writing
  • Writing basic MapReduce programs
    1. Constructing the basic template of a MapReduce program
    2. Counting things
    3. Adapting for Hadoop’s API changes
    4. Streaming in Hadoop
    5. Improving performance with combiners
  • Advanced MapReduce
    1. Chaining MapReduce jobs
    2. Joining data from different sources
    3. Creating a Bloom filter
  • Programming Practices
    1. Developing MapReduce programs
    2. Monitoring and debugging on a production cluster
    3. Tuning for performance
  • Cookbook
    1. Passing job-specific parameters to your tasks
    2. Probing for task-specific information
    3. Partitioning into multiple output files
    4. Inputting from and outputting to a database
    5. Keeping all output in sorted order
  • Managing Hadoop
    1. Setting up parameter values for practical use
    2. Checking system’s health
    3. Setting permissions
    4. Managing quotas
    5. Enabling trash
    6. Removing DataNodes
    7. Adding DataNodes
    8. Managing NameNode and Secondary NameNode
    9. Recovering from a failed NameNode
    10. Designing network layout and rack awareness
    11. Scheduling jobs from multiple users
  • Running Hadoop in the cloud
    1. Introducing Amazon Web Services
    2. Setting up AWS
    3. Setting up Hadoop on EC2
    4. Running MapReduce programs on EC2
    5. Cleaning up and shutting down your EC2 instances
    6. Amazon Elastic MapReduce and other AWS services
  • Programming with Pig
    1. Installing Pig
    2. Running Pig
    3. Learning Pig Latin through Grunt
    4. Speaking Pig Latin
    5. Working with user-defined functions
    6. Working with scripts
    7. Seeing Pig in action—example of computing similar patents
  • Hadoop Related Technologies
    1. Hive
    2. Apache Accumulo
    3. Other Hadoop-related stuff
  • Case studies
    1. Converting 11 million image documents from the New York Times archive
    2. Mining data at China Mobile
    3. Recommending the best websites at StumbleUpon
    4. Building analytics for enterprise search—IBM’s Project ES2