Benefits of Big Data Processing

Ability to process 'Big Data' brings in multiple benefits, such as-
• Businesses can utilize outside intelligence while taking decisions
Access to social data from search engines and sites like facebook, twitter are enabling organizations to fine tune their business strategies.

• Improved customer service
Traditional customer feedback systems are getting replaced by new systems designed with 'Big Data' technologies. In these new systems, Big Data and natural language processing technologies are being used to read and evaluate consumer responses.

• Early identification of risk to the product/services, if any
 
• Better operational efficiency
'Big Data' technologies can be used for creating staging area or landing zone for new data before identifying what data should be moved to the data warehouse. In addition, such integration of 'Big Data' technologies and data warehouse helps organization to offload infrequently accessed data.

What is Hadoop? – Simplified!

Scenario 1: Any global bank today has more than 100 Million customers doing billions of transactions every month

Scenario 2: Social network websites or eCommerce websites track customer behaviour on the website and then serve relevant information / product.

Traditional systems find it difficult to cope up with this scale at required pace in cost-efficient manner.

This is where Big data platforms come to help. In this article, we introduce you to the mesmerizing world of Hadoop. Hadoop comes handy when we deal with enormous data. It may not make the process faster, but gives us the capability to use parallel processing capability to handle big data. In short, Hadoop gives us capability to deal with the complexities of high volume, velocity and variety of data (popularly known as 3Vs).

Please note that apart from Hadoop, there are other big data platforms e.g. NoSQL (MongoDB being the most popular), we will take a look at them at a later point.

Introduction to Hadoop

Hadoop is a complete eco-system of open source projects that provide us the framework to deal with big data. Let’s start by brainstorming the possible challenges of dealing with big data (on traditional systems) and then look at the capability of Hadoop solution.

Following are the challenges I can think of in dealing with big data :

1. High capital investment in procuring a server with high processing capacity.

2. Enormous time taken

3. In case of long query, imagine an error happens on the last step. You will waste so much time making these iterations.

4. Difficulty in program query building

Here is how Hadoop solves all of these issues :

LinuxWorld Informatics Pvt. Ltd Offer Bigdata hadoop Training

Hadoop turns 10, Big Data industry rolls along

Apache Hadoop, the open source project that arguably sparked the Big Data craze, turned 10 years old this week. The project's founder, Cloudera's Doug Cutting, waxed nostalgic as vendors in the space churned out new releases of their own.

It's hard to believe, but it's true. The Apache Hadoop project, the open source implementation of Google's File System (GFS) and MapReduce execution engine, turned 10 this week.

The technology, originally part of Apache Nutch, an even older open source project for Web crawling, was separated out into its own project in 2006, when a team at Yahoo was dispatched to accelerate its development.

Proud dad weighs inDoug Cutting, founder of both projects (as well as Apache Lucene), formerly of Yahoo, and presently Chief Architect at Cloudera, wrote a blog post commemorating the birthday of the project, named after his son's stuffed elephant toy.

In his post, Cutting correctly points out that "Traditional enterprise RDBMS software now has competition: open source, big data software." The database industry had been in real stasis for well over a decade. Hadoop and NoSQL changed that, and got the incumbent vendors off their duffs and back in the business of refreshing their products with major new features

Sleeping giants awaken
Microsoft SQL Server now supports columnstore indexes in order to handle analytic queries on large volumes of data and its upcoming 2016 version adds PolyBase functionality for integrated query of data in Hadoop. Meanwhile, Oracle and IBM have added their own Hadoop bridges, along with better handling of semi-structured data.

Teradata has pivoted rather sharply towards Hadoop and Big Data, starting with its acquisition of Aster Data and continuing through its multifaceted partnerships with Cloudera and Hortonworks. Meanwhile, in the Hadoop Era, perhaps in deference to Teradata, virtually every megavendor acquired one of the data warehousing pure plays.

New generationCutting points out, also accurately, that the original core components of Hadoop have been challenged and/or replaced: "New execution engines like Apache Spark and new storage systems like Apache Kudu (incubating) demonstrate that this software ecosystem evolves rapidly, with no central point of control." Granted, both of these projects are heavily championed by Cloudera, so take the commentary with a grain of salt.

Salt or no salt though, Cutting's comment that the Hadoop ecosystem has "no central point of control" is one worth considering carefully; because, while it is correct, it's not necessarily good. The term "creative destruction" sometimes truly is an oxymoron. The Big Data scene's rapid technology replacement cycles leave the space stability-challenged.

Give peace a chancePerhaps, but the moving technology target may also mean they get no software at all, because the current environment is sufficiently risk-prone as to hinder the growth of enterprise projects. We need some equilibrium if we want growth to be proportionate to the level of technological innovation.

Cutting concludes his post by declaring: "I look forward to following Hadoop's continued impact as the data century unfolds." While I'm not sure data and analytics will define the whole century, they probably have a good decade or two. Hopefully the industry can get a little better at developing standards that are cooperative and compatible, rather than overlapping and competitive. We don't want to go back to stasis, but more navigable terrain would suit the industry and its customers

Meanwhile, back in the competitive marketSpeaking of the industry, there were a slew of announcements this week, beside (and even despite) Hadoop's birthday.:

• Pentaho introduced Python language integration into its Data Integration Suite
• Paxata launched its new Winter '15 release (albeit in 2016), which includes new auto number and fill down transformations, new algorithms to aid its data prep recommendations, and integration with LDAP and SAML, for enterprise security, single sign-on and identity management
• SkyTree, a predictive analytics vendor, discussed that it will soon launch a free single-user version of its product, which it will soon announce more formally (and RapidMiner, also in the predictive space, released its new version 7 last week, with a revamped UI)
• NoSQL vendor Aerospike launched a new release of its eponymous database, which now features geospatial data support, added resiliency in cloud-hosted environments and server-side support for list and map data structures

Weekend pondering
That's a pretty busy week. And I dare say, without Hadoop as a catalyst, it would have been much less so. As climate change, financial markets, geopolitics and the price of oil reach frightening new levels of volatility, the data sector of the technology industry is thriving. We might hope that the technology around Big Data could be deployed to help solve, or at least better understand, some of our world's truly big problems.
This won't be the century of data unless that in fact happens

Article Source - http://www.zdnet.com/article/hadoop-turns-10-big-data-industry-rolls-along/

6 Reasons Why Java Developers Should Learn Hadoop

Imagine there are two girls standing in front of you - The first girl is cute, beautiful, interesting and has the smile that any guy would die for. And the other girl is average-looking, quiet, not-so-impressive... no different from the ones that you usually see in the restaurant cash counter. Which girl will you call out for a date? If you're like me, you will choose the attractive girl. You see, life is full of options and making the right choice is what matters the most.

If you're a Java developer, then you probably have more choices to make - like the switch from Java to Hadoop.

Big data and Hadoop are the two most popular buzzwords in the industry. Chances are that you have come across these two terms on the Java payscale forums or seen your senior colleagues making the switch to get bigger paychecks. I'll tell you what, the upgrade from Java to Hadoop is not just about staying updated with the latest technology or getting appraisals - it's about being competent and putting your career on the fifth gear.

The good news for all the aspiring Hadoop developers is that, the Big Data industry has already crossed the $50 billion dollar mark and over 64% of the top 720 companies worldwide are interesting to invest in this forward-thinking technology as revealed by Gartner in 2013.

If that's not convincing, then take a look at these stats:

1. According to an IDC report, the Big Data industry is growing at the rate of 31.7% per year.
2. Java developers are seen as the best replacement option for Hadoop developers, says Forrester.
3. Hadoop developers enjoy a mighty 250% pay hike than Java developers, as stated in an Analytics Industry Report.

What's special about Hadoop?

Unlike the traditional databases which weren't capable of dealing with large volumes of data, Hadoop offers the quickest, cheapest, and smartest way to store and process giant volumes of data - and that's the reason why it is so popular among big corporations, government organizations, hospitals, universities, financial services, marketing agencies, etc. The best way to familiarize with the language is to check out a beginner's big data hadoop course.

Okay, now let's some reasons why Java developers should switch to Hadoop.

1. Easy To Learn For Java Developers

A tennis player like Rafael Nadal loves clay courts because the surface suits him well and that's where he has been most successful. Similarly, any Java developer would love Hadoop because it's completely written in Java - a language that you are already so familiar with. Switching from Java to Hadoop is a cake-walk for professionals like you because the MapReduce script used in the Hadoop is actually written in Java itself. Awesome, isn't it?

Your Java skills will come in handy when debugging Hadoop apps and employing Pig (programming tool) Latin commands.

2. Helps You To Stay Ahead Of Your Competition

If you are a Java professional, you are just seen as a person in the crowd. But, if you are a Hadoop developer, you are seen as potential leader in the crowd. Big Data and Hadoop jobs are a hot deal in the market and Java professionals with the required skill set are easily picked by big companies for high salary packages. All you have to do is attend a big data hadoop  training program and learn the concepts from an expert.

3. Scope To Move Into Bigger Domains

Fortunately for you, the road doesn't end with Hadoop and MapReduce. There is always the golden opportunity to use your Hadoop skills and expertise to move into higher levels such as Artificial Intelligence, Data Science, Sensor Web data, and Machine Learning. These are emerging markets, and you'll see them dominate the industry in the next 4-5 years. Good knowledge in Big Data and Hadoop could boost your chances of getting into some of the bigger Big Data-dependent companies such as Amazon, Yahoo, Facebook, Twitter, IBM, and eBay.

4. Lucrative Packages For Hadoop Professionals

By switching from Java to Hadoop, you can expect a higher salary and better career prospects - the kind of salary and designation that your wife would like to rave about. According to Indeed, the average salary for a Big Data Hadoop developer with 1-2 years of experience is around $140,000 per annum in the United States. However, as you gain experience and become a senior Hadoop developer, you will be able to make a good $400,000+ salary.

5. An Improved Quality Of Work

Learning Big Data Hadoop can be highly beneficial because it will help you to deal with bigger, complex projects much easier and deliver better output than your colleagues. In order to be considered for appraisals, you need to be someone who can make a difference in the team, and that's what Hadoop lets you to be.

6. Grow With The Industry

With IDC predicting that the Big Data and Hadoop user base (big companies and government organizations) is likely to increase at 27% per year, you have a great opportunity to upgrade your knowledge and skills and grow with the industry.

Big Data and Hadoop are widely used in applications such as IT log analytics, Fraud detection, Social media analysis, and Call centre analytics - and learning a big data hadoop tutorial could be the way to kick-start your Hadoop career right away. Once you do that, you will find that staying updated with the latest technology will be a lot easier and getting into top organizations will never be 'just a dream' - it will be a reality.

That's about it, folks! These are some rock-solid reasons why learning Hadoop is important and how it can help take your career to the next level

Article Source: http://EzineArticles.com/9189164

Overview of Hadoop Applications

Hadoop is nothing but a source of software framework that is generally used in the processing immense and bulk data simultaneously across many servers. In the recent years, it has turned out to be one of most viable option for enterprises, which has the never-ending requirement to save and manage all the data. Web based businesses such as Facebook, Amazon, eBay, and Yahoo have used high-end Hadoop applications to manage their large data sets. It is believed that Hadoop Training is still relevant to both small organizations as well as big time businesses.

Hadoop is able to process a huge chunk of data in a lesser time which enabled the companies to analyze that this was not possible before within that stipulated time. Another important advantage of the Hadoop applications is the cost effectiveness, which cannot be availed in any other technologies. One can avoid the high cost involved in the software licenses and the fees that has to be upgraded periodically when using anything apart from Hadoop. It is highly recommended for businesses, which have to work with huge amount of data, to go for Hadoop applications as it helps in fixing any issues.

Actually, Hadoop applications are made up of two parts; one is the HDFS, which means the Hadoop Distributed File System while the other is the Hadoop map reduce that helps in the processing of data and scheduling of job depending upon the priority, which is a technique that initially originated in Google search engine. Along with these two primary components, there are nine other parts, which are decided as per the distribution one uses along with other complementary tools. There are three most common functions of Hadoop applications. The first function is the storage and analysis of all the data, which does not require the loading of the relational database management system. Secondly, it is used in the conversion of huge repository of semi-structured and unstructured data, for example a log file in the form of a structured data. Such complicated data are hard to understand in SQL tools like analyzing the graph and data mining.

Hadoop applications are mostly used in the web-related businesses wherein one has to work with big log files and data from the social network sites. When it comes to media or the advertising world, enterprises use Hadoop, which enables the best performance of ad offer analysis and help understand online reviews. Before using any Hadoop tool, it is advisable to read through the Hadoop map tutorials available online.

Article Source: http://EzineArticles.com/6531213

PMO using Big Data techniques on mygov.in to translate popular mood into government action

NEW DELHI: The Prime Minister's Office is using Big Data techniques to process ideas thrown up by citizens on its crowd sourcing platform mygov. in, place them in context of the popular mood as reflected in trends on social media, and generate actionable reports for ministries and departments to consider and implement.

The Modi government has roped in global consulting firm PwC to assist in the data mining exercise, and now wants to elevate Mygov.in platform from a one-way flow of citizens' ideas to a dialogue where the government keeps them abreast of some of the actions that emerge from their brainstorming.

"There is a large professional data analytics team working behind the scenes to process and filter key points emerging from debates on mygov.in, gauge popular mood about particular issues from social media sites like Twitter and Facebook," said a senior official aware of the development, adding that these are collated into special reports about possible action points that are shared with the PMO and line ministries. Ministries are being asked to revert with an action taken report on these ideas and policy suggestions currently being generated on 19 different policy challenges such as expenditure reforms, job creation, energy conservation, skill development and government initiatives such as Clean India, Digital India and Clean Ganga.

With the PM inviting Indian communities in America and Australia to join the online platform, which he has termed a 'mass movement towards Surajya', the traffic handling capacity of mygov.in is being scaled up consistently, the official said.

PwC executive director Neel Ratan said that the firm is 'helping the government' process the citizen inputs coming through on Mygov. in in.

"There is a science and art behind it. We have people constantly looking at all ideas coming up, filtering them and after a lot of analysis, correlating it to sentiments coming through on the rest of social media," he said, stressing this is throwing up interesting trends and action points, being relayed to ministries. "It's turning out to be fairly action-oriented. I think it is distinctly possible that 30-50 million people would be actively contributing to Mygov.in over the next year and a half, given its current pace of growth," Ratan said.

PwC's global leader in government and public services Jan Sturesson told ET the participative governance model being adopted through mygov.in could become a model for the developed world.

"The biggest issue for governments today is how to be relevant. If all citizens are treated with dignity and invited to collaborate, it can be easier for administrations to have a direct finger on the pulse of the nation rather than lose it in transmission through multiple layers of bureaucracy," he said, not ruling out the possibility of using the mygov.in for quick referendums on contemporary policy dilemmas in a couple of years.
"The problem in the West has been that the US, Australia and UK follow a public management philosophy that treats citizens as consumers. That's ridiculous, because a consumer pays the bill and complains, while a citizen engages differently and takes responsibility," said Sturesson.

Within the 19 broad citizen engagement themes on mygov.in, there are multiple discussion groups focused on specific subsectors and themes. When it was launched in July, the site enabled brainstorming among its registered users around seven policy challenges. Users are allowed to sign up for four discussion groups in areas of interest apart from a group dealing with issues in their immediate vicinity.

Article Source - http://articles.economictimes.indiatimes.com/2014-11-26/news/56490626_1_mygov-digital-india-modi-government

Learning Hadoop with Linux World India

The best Big Data Hadoop Training in Jaipur is offered by LinuxWorld India. When it comes to technological training, we are the best in the business. The organization was established in 2005. Our training comprises of all the expert professionals and teachers who impart tremendous knowledge. When it comes to Cisco certifications, we are the first preference. We bring together a blend of network training and solutions with the help of our authorized training partner. We believe in providing the knowledge to its betterment. With the best study in Hadoop, you can build your super computer. We provide the best classroom and training of Version 2 Hadoop. We are the first ones to bring this facility to India.

The training fees required for this course is Rs. 25500 and the course module will be delivered to you by us. After completing this course with us you will be able a master in Hadoop and will produce a framework of MapReduce. You will also be an expert in writing complex programs related to MapReduce. Hadoop is a tool that was built on Java, and it focuses on improving the performance of hardware. By studying this, you can also create a cluster of data that will help you to program different models.

Remember FLURPS to design better big data analytic solutions

FLURPS is an acronym for six components of well-rounded big data designs: Functionality, Localizability, Usability, Reliability, Performance, Supportability. Here's the case for using this template.

I've been advocating for customer-centric design as long as I've been designing solutions for customers. I still do this, because I have to. It's remarkable to me that after decades of building high-tech solutions for customers, technologists still seem to build solutions in an IT vacuum and then get upset when customers don't find them very functional.

Gold plating is a term used to describe developers who infer customer requirements and subsequently build features that the end users never requested -- because the developers know better. Unfortunately, data scientists are carrying this tradition forward with analytic solutions. That said, I wouldn't categorically dismiss any requirement that doesn't come from a customer or end user. It may seem odd coming from such a strong advocate of customer-centric design, but there are aspects of a well-built solution that customers don't know or appreciate.

When designing a big data analytic solution, make sure it includes a well-rounded set of requirements, including ones that the end user won't directly know about or care about.

What's FLURPS?

FLURPS is a great acronym that I learned as a young computer Big Data Training in Jaipur, and it works great as a template for building well-rounded analytic solutions. FLURPS stands for: Functionality, Localizability, Usability, Reliability, Performance, Supportability. It seems like a lost acronym that I'd like to resurrect to help us design and build better solutions.

This funny-sounding acronym reminds me of a big, hairy puppet like Mr. Snuffleupagus -- that's why it has stuck in my mind for so many years. Let's go through the different elements of FLURPS and how they can enhance your design.

Functionality

Functionality remains the key component of design; it represents all the features the customer knows and wants. When building a requirements document, most analysts separate functional from non-functional requirements, which is a good practice. Furthermore, functional requirements should always take precedence over non-functional requirements. Never sacrifice functional requirements for non-functional requirements -- you should satisfy non-functional requirements in addition to functional requirements.

Localizability

Localizability handles geographical concerns such as language. Internationalization (or i18n, for those in the know) is closely related to localizability in that it architecturally provides the technical infrastructure to localize a solution. Knowing that your recommendation engine will be used globally, you may internationalize it by automatically sensing where your user is located, and then localize it by providing, for example, German-, Russian-, or Chinese-specific content. Bear in mind that good localizability extends beyond language translation and caters to cultural differences in functionality.

Usability

Usability deals with the customer experience. This is a pet peeve of mine -- I'm tired of seeing analytic solutions that force the user into the mind of the developer. For instance, it's very common to see use cases where a batch operation seems obvious, but the solution only allows single transaction processing. If I could possibly have hundreds of input variables to my predictive analytics engine, why should I have to create them one by one?

Although usability seems like it should fall into the functional category, it does not. Most customers don't know how to design a usable solution; however, they know when it's not usable. Putting a user experience expert on your team is a fantastic idea.

Reliability

Reliability handles the stability of your application. Reliability is not something end users contemplate because they assume your solution will be stable; when it's not, frustration can quickly escalate to extreme dissatisfaction.

You must build reliable solutions. How many times have you lost work because your system crashed? And by the way, a cute little icon telling you that the system crashed doesn't help. Build requirements into your solution to recover from exceptional situations and gracefully exit only when all possible routes of recovery are lost. I once designed a web application that went through four or five levels of exception before it finally, gracefully quit -- after saving all of the user's work. The users never knew the application was going into its third and fourth level of exception, and that's the way it should be.

Performance

Performance is a bigger deal than you might think. I recently trained a group of users on a new web-based system that would on occasion take several minutes for a submenu to appear -- the industry standard is between two and three seconds. The performance of the system broadcasted the quality of the rest of the system, and it wasn't positive.

I know performance issues can be difficult to track down, but that's your problem, not the users. Make sure reasonable response times are documented in your requirements and thoroughly tested when the system is built.

Supportability

Supportability is the last, but not the least important, component of a robust design. Whether you're designing a product that will be used by customers or an internal system that will be used by employees, it's vitally important that the operations group is in a good position to support the solution.

For analytic solutions, supportability often extends beyond requirements into organizational design. The instrumentation on an analytic solution is often sophisticated, so it's important to staff the operations function with very knowledgeable technicians -- maybe even other data scientists. When I'm putting together a development team, I often include at least one person from the support team; this way, they can influence the solution's design from the perspective of someone who's going to support it.

Article Source - http://www.techrepublic.com/article/remember-flurps-to-design-better-big-data-analytic-solutions/

The Google Cloud Platform: 10 things you need to know

The Google Cloud Platform comprises many of Google's top tools for developers. Here are 10 things you might not know about it.


The infrastructure-as-a-service (IaaS) market has exploded in recent years. Google stepped into the fold of IaaS providers, somewhat under the radar. The Google Cloud Platform is a group of cloud computing tools for developers to build and host web applications.

It started with services such as the Google App Engine and quickly evolved to include many other tools and services. While the Google Cloud Platform was initially met with criticism of its lack of support for some key programming languages, it has added new features and support that make it a contender in the space.

Here's what you need to know about the Google Cloud Platform.

1. Pricing

Google recently shifted its pricing model to include sustained-use discounts and per-minute billing. Billings starts with a 10-minute minimum and bills per minute for the following time. Sustained-use discounts begin after a particular instance is used for more than 25% of a month. Users receive a discount for each incremental minute used after they reach the 25% mark. Developers can find more information here.

If you're wondering what it would cost for your organization, try Google's pricing calculator.

2. Cloud Debugger
The Cloud Debugger gives developers the option to assess and debug code in production. Developers can set a watchpoint on a line of code, and any time a server request hits that line of code, they will get all of the variables and parameters of that code. According to Google blog post, there is no overhead to run it and "when a watchpoint is hit very little noticeable performance impact is seen by your users."

3. Cloud Trace
Cloud Trace lets you quickly figure out what is causing a performance bottleneck and fix it. The base value add is that it shows you how much time your product is spending processing certain requests. Users can also get a report that compares performances across releases.

4. Cloud Save

The Cloud Save API was announced at the 2014 Google I/O developers conference by Greg DeMichillie, the director of product management on the Google Cloud Platform. Cloud Save is a feature that lets you "save and retrieve per user information." It also allows cloud-stored data to be synchronized across devices.

5. Hosting
The Cloud Platform offers two hosting options: the App Engine, which is their Platform-as-a-Service and Compute Engine as an Infrastructure-as-a-Service. In the standard App Engine hosting environment, Google manages all of the components outside of your application code.

The Cloud Platform also offers managed VM environments that blend the auto-management of App Engine, with the flexibility of Compute Engine VMs.The managed VM environment also gives users the ability to add third-party frameworks and libraries to their applications.

6. Andromeda
Google Cloud Platform networking tools and services are all based on Andromeda, Google's network virtualization stack. Having access to the full stack allows Google to create end-to-end solutions without compromising functionality based on available insertion points or existing software.

According to a Google blog post, "Andromeda is a Software Defined Networking (SDN)-based substrate for our network virtualization efforts. It is the orchestration point for provisioning, configuring, and managing virtual networks and in-network packet processing."

7. Containers
Containers are especially useful in a PaaS situation because they assist in speeding deployment and scaling apps. For those looking for container management in regards to virtualization on the Cloud Platform, Google offers its open source container scheduler known as Kubernetes. Think of it as a Container-as-a-Service solution, providing management for Docker containers.

8. Big Data
The Google Cloud Platform offers a full big data solution, but there are two unique tools for big data processing and analysis on Google Cloud Platform. First, BigQuery allows users to run SQL-like queries on terabytes of data. Plus, you can load your data in bulk directly from your Google Cloud Storage.

The second tool is Google Cloud Dataflow. Also announced at I/O, Google Cloud Dataflow allows you to create, monitor, and glean insights from a data processing pipeline. It evolved from Google's MapReduce.

9. Maintenance
Google does routine testing and regularly send patches, but it also sets all virtual machines to live migrate away from maintenance as it is being performed.

"Compute Engine automatically migrates your running instance. The migration process will impact guest performance to some degree but your instance remains online throughout the migration process. The exact guest performance impact and duration depend on many factors, but it is expected most applications and workloads will not notice," the Google developer website said.

VMs can also be set to shut down cleanly and reopen away from the maintenance event.

10. Load balancing
In June, Google announced the Cloud Platform HTTP Load Balancing to balance the traffic of multiple compute instances across different geographic regions.

To more about Big Data Hadoop Training in Jaipur please Visit on --

http://www.bigdatahadoop.info/

To More Visit - http://www.techrepublic.com/article/the-google-cloud-platform-10-things-you-need-to-know/

The Early Release Books Keep Coming: This Time, Hadoop Security

We are thrilled to announce the availability of the early release of Hadoop Security, a new book about security in the Apache Hadoop ecosystem published by O’Reilly Media. The early release contains two chapters on System Architecture and Securing Data Ingest and is available in O’Reilly’s catalog and in Safari Books.

Hadoop security

The goal of the book is to serve the experienced security architect that has been tasked with integrating Hadoop into a larger enterprise security context. System and application administrators also benefit from a thorough treatment of the risks inherent in deploying Hadoop in production and the associated how and why of Hadoop security.

As Hadoop continues to mature and become ever more widely adopted, material must become specialized for the security architects tasked with ensuring new applications meet corporate and regulatory policies. While it is up to operations staff to deploy and maintain the system, they won’t be responsible for determining what policies their systems must adhere to. Hadoop is mature enough that dedicated security professionals need a reference to navigate the complexities of security on such a massive scale. Additionally, security professionals must be able to keep up with the array of activity in the Hadoop security landscape as exemplified by new projects like Apache Sentry (incubating) and cross-project initiatives such as Project Rhino.

Security architects aren’t interested in how to write a MapReduce job or how HDFS splits files into data blocks, they care about where data is going and who will be able to access it. Their focus is on putting into practice the policies and standards necessary to keep their data secure. As more corporations turn to Hadoop to store and process their most valuable data, the risks with a potential breach of those systems increases exponentially. Without a thorough treatment of the subject, organizations will delay deployments or resort to siloed systems that increase capital and operating costs.

The first chapter available is on the System Architecture where Hadoop is deployed. It goes into the different options for deployment: in-house, cloud, and managed. The chapter also covers how major components of the Hadoop stack get laid out physically from both a server perspective and a network perspective. It gives a security architect the necessary background to put the overall security architecture of a Hadoop deployment into context.

The second available chapter is on Securing Data Ingest it covers the basics of Confidentiality, Integrity, and Availability (CIA) and applies them to feeding your cluster with data from external systems. In particular, the two most common data ingest tools, Apache Flume and Apache Sqoop, are evaluated for their support of CIA. The chapter details the motivation for securing your ingest pipeline as well as providing ample information and examples on how to configure these tools for your specific needs. The chapter also puts the security of your Hadoop data ingest flow into the broader context of your enterprise architecture.

We encourage you to take a look and get involved early. Security is a complex topic and it never hurts to get a jump start on it. We’re also eagerly awaiting feedback. We would never have come this far without the help of some extremely kind reviewers. You can also expect more chapters to come in the coming months. We’ll continue to provide summaries on this blog as we release new content so you know what to expect.

If anyone want to learn Big Data Hadoop Training than Visit on - http://www.bigdatahadoop.info