Tuesday, 16 December 2014

Overview of Hadoop Applications

Hadoop is nothing but a source of software framework that is generally used in the processing immense and bulk data simultaneously across many servers. In the recent years, it has turned out to be one of most viable option for enterprises, which has the never-ending requirement to save and manage all the data. Web based businesses such as Facebook, Amazon, eBay, and Yahoo have used high-end Hadoop applications to manage their large data sets. It is believed that Hadoop Training is still relevant to both small organizations as well as big time businesses.
Hadoop is able to process a huge chunk of data in a lesser time which enabled the companies to analyze that this was not possible before within that stipulated time. Another important advantage of the Hadoop applications is the cost effectiveness, which cannot be availed in any other technologies. One can avoid the high cost involved in the software licenses and the fees that has to be upgraded periodically when using anything apart from Hadoop. It is highly recommended for businesses, which have to work with huge amount of data, to go for Hadoop applications as it helps in fixing any issues.

Actually, Hadoop applications are made up of two parts; one is the HDFS, which means the Hadoop Distributed File System while the other is the Hadoop map reduce that helps in the processing of data and scheduling of job depending upon the priority, which is a technique that initially originated in Google search engine. Along with these two primary components, there are nine other parts, which are decided as per the distribution one uses along with other complementary tools. There are three most common functions of Hadoop applications. The first function is the storage and analysis of all the data, which does not require the loading of the relational database management system. Secondly, it is used in the conversion of huge repository of semi-structured and unstructured data, for example a log file in the form of a structured data. Such complicated data are hard to understand in SQL tools like analyzing the graph and data mining.

Hadoop applications are mostly used in the web-related businesses wherein one has to work with big log files and data from the social network sites. When it comes to media or the advertising world, enterprises use Hadoop, which enables the best performance of ad offer analysis and help understand online reviews. Before using any Hadoop tool, it is advisable to read through the Hadoop map tutorials available online.

Friday, 28 November 2014

PMO using Big Data techniques on mygov.in to translate popular mood into government action

NEW DELHI: The Prime Minister's Office is using Big Data techniques to process ideas thrown up by citizens on its crowd sourcing platform mygov. in, place them in context of the popular mood as reflected in trends on social media, and generate actionable reports for ministries and departments to consider and implement.
The Modi government has roped in global consulting firm PwC to assist in the data mining exercise, and now wants to elevate Mygov.in platform from a one-way flow of citizens' ideas to a dialogue where the government keeps them abreast of some of the actions that emerge from their brainstorming.

"There is a large professional data analytics team working behind the scenes to process and filter key points emerging from debates on mygov.in, gauge popular mood about particular issues from social media sites like Twitter and Facebook," said a senior official aware of the development, adding that these are collated into special reports about possible action points that are shared with the PMO and line ministries. Ministries are being asked to revert with an action taken report on these ideas and policy suggestions currently being generated on 19 different policy challenges such as expenditure reforms, job creation, energy conservation, skill development and government initiatives such as Clean India, Digital India and Clean Ganga.

With the PM inviting Indian communities in America and Australia to join the online platform, which he has termed a 'mass movement towards Surajya', the traffic handling capacity of mygov.in is being scaled up consistently, the official said.

PwC executive director Neel Ratan said that the firm is 'helping the government' process the citizen inputs coming through on Mygov. in in.

"There is a science and art behind it. We have people constantly looking at all ideas coming up, filtering them and after a lot of analysis, correlating it to sentiments coming through on the rest of social media," he said, stressing this is throwing up interesting trends and action points, being relayed to ministries. "It's turning out to be fairly action-oriented. I think it is distinctly possible that 30-50 million people would be actively contributing to Mygov.in over the next year and a half, given its current pace of growth," Ratan said.

PwC's global leader in government and public services Jan Sturesson told ET the participative governance model being adopted through mygov.in could become a model for the developed world.

"The biggest issue for governments today is how to be relevant. If all citizens are treated with dignity and invited to collaborate, it can be easier for administrations to have a direct finger on the pulse of the nation rather than lose it in transmission through multiple layers of bureaucracy," he said, not ruling out the possibility of using the mygov.in for quick referendums on contemporary policy dilemmas in a couple of years.
"The problem in the West has been that the US, Australia and UK follow a public management philosophy that treats citizens as consumers. That's ridiculous, because a consumer pays the bill and complains, while a citizen engages differently and takes responsibility," said Sturesson.
Within the 19 broad citizen engagement themes on mygov.in, there are multiple discussion groups focused on specific subsectors and themes. When it was launched in July, the site enabled brainstorming among its registered users around seven policy challenges. Users are allowed to sign up for four discussion groups in areas of interest apart from a group dealing with issues in their immediate vicinity.

Article Source - http://articles.economictimes.indiatimes.com/2014-11-26/news/56490626_1_mygov-digital-india-modi-government

Saturday, 22 November 2014

Learning Hadoop with Linux World India



The best Big Data Hadoop Training in Jaipur is offered by LinuxWorld India. When it comes to technological training, we are the best in the business. The organization was established in 2005. Our training comprises of all the expert professionals and teachers who impart tremendous knowledge. When it comes to Cisco certifications, we are the first preference. We bring together a blend of network training and solutions with the help of our authorized training partner. We believe in providing the knowledge to its betterment. With the best study in Hadoop, you can build your super computer. We provide the best classroom and training of Version 2 Hadoop. We are the first ones to bring this facility to India.

The training fees required for this course is Rs. 25500 and the course module will be delivered to you by us. After completing this course with us you will be able a master in Hadoop and will produce a framework of MapReduce. You will also be an expert in writing complex programs related to MapReduce. Hadoop is a tool that was built on Java, and it focuses on improving the performance of hardware. By studying this, you can also create a cluster of data that will help you to program different models.

Thursday, 13 November 2014

Big Data Training in Jaipur



Intelligence is a leading an advanced Hadoop Training in Jaipur. They are especially known for the Data Warehousing and Hadoop in Jaipur. It includes the core competencies such as ABAP, DW, DI, HANA, SAP BO, AIX, Solaris, Red Hat Certification, Linux, PLSQL, Oracle SQL, Business Objects, Data stage, Informatics, Congos, Software Testing, ISTQB certification, Amazon Web Services, Mobile application of the developer using the IOS and Android. They also provide you with the best online training for all of the above technologies. Their main objective is to provide production based knowledge for every training module. They also have experienced trainers and well-equipped lab for every technology. 
The Hadoop administrators are one of the finest and famous among the world that are more in demand and are highly compensated in the technical role. Many of them are nowadays going for the CCAH qualification. They get to learn all the Hadoop topics such as first, to determine the infrastructure for your cluster and to correct the hardware. Second, learn the internals of the HDFS, Map Reduce and YARN. Third, learn the deployment to integrate with the data center and proper cluster configuration. Fourth, learn how to load data into the cluster from the RDBMS with the help of Scoop and from the dynamically generated files with the help of the Flume. 
Fifth, to configure the fair scheduler in order to provide service level agreements for the multiple users of the cluster. Sixth, solving the Hadoop issues, tuning, diagnosing and troubleshooting problems. Seventh, best practices for maintaining and preparing the production in the Apache Hadoop. This training helps you with good jobs, you get the complete technology specified, and they conduct campus drives. This course is best for the IT managers and the system administrators who have the basic knowledge about the Linux experience. Despite the Apache Hadoop, knowledge is not required.

To More about the Big Data Hadoop Training in Jaipur ,Please Click This

Friday, 31 October 2014

Remember FLURPS to design better big data analytic solutions

FLURPS is an acronym for six components of well-rounded big data designs: Functionality, Localizability, Usability, Reliability, Performance, Supportability. Here's the case for using this template.
bigdata082613.gif
I've been advocating for customer-centric design as long as I've been designing solutions for customers. I still do this, because I have to. It's remarkable to me that after decades of building high-tech solutions for customers, technologists still seem to build solutions in an IT vacuum and then get upset when customers don't find them very functional.
Gold plating is a term used to describe developers who infer customer requirements and subsequently build features that the end users never requested -- because the developers know better. Unfortunately, data scientists are carrying this tradition forward with analytic solutions. That said, I wouldn't categorically dismiss any requirement that doesn't come from a customer or end user. It may seem odd coming from such a strong advocate of customer-centric design, but there are aspects of a well-built solution that customers don't know or appreciate.
When designing a big data analytic solution, make sure it includes a well-rounded set of requirements, including ones that the end user won't directly know about or care about.

What's FLURPS?

FLURPS is a great acronym that I learned as a young computer Big Data Training in Jaipur, and it works great as a template for building well-rounded analytic solutions. FLURPS stands for: Functionality, Localizability, Usability, Reliability, Performance, Supportability. It seems like a lost acronym that I'd like to resurrect to help us design and build better solutions.
This funny-sounding acronym reminds me of a big, hairy puppet like Mr. Snuffleupagus -- that's why it has stuck in my mind for so many years. Let's go through the different elements of FLURPS and how they can enhance your design.

Functionality

Functionality remains the key component of design; it represents all the features the customer knows and wants. When building a requirements document, most analysts separate functional from non-functional requirements, which is a good practice. Furthermore, functional requirements should always take precedence over non-functional requirements. Never sacrifice functional requirements for non-functional requirements -- you should satisfy non-functional requirements in addition to functional requirements.

Localizability

Localizability handles geographical concerns such as language. Internationalization (or i18n, for those in the know) is closely related to localizability in that it architecturally provides the technical infrastructure to localize a solution. Knowing that your recommendation engine will be used globally, you may internationalize it by automatically sensing where your user is located, and then localize it by providing, for example, German-, Russian-, or Chinese-specific content. Bear in mind that good localizability extends beyond language translation and caters to cultural differences in functionality.

Usability

Usability deals with the customer experience. This is a pet peeve of mine -- I'm tired of seeing analytic solutions that force the user into the mind of the developer. For instance, it's very common to see use cases where a batch operation seems obvious, but the solution only allows single transaction processing. If I could possibly have hundreds of input variables to my predictive analytics engine, why should I have to create them one by one?
Although usability seems like it should fall into the functional category, it does not. Most customers don't know how to design a usable solution; however, they know when it's not usable. Putting a user experience expert on your team is a fantastic idea.

Reliability

Reliability handles the stability of your application. Reliability is not something end users contemplate because they assume your solution will be stable; when it's not, frustration can quickly escalate to extreme dissatisfaction.
You must build reliable solutions. How many times have you lost work because your system crashed? And by the way, a cute little icon telling you that the system crashed doesn't help. Build requirements into your solution to recover from exceptional situations and gracefully exit only when all possible routes of recovery are lost. I once designed a web application that went through four or five levels of exception before it finally, gracefully quit -- after saving all of the user's work. The users never knew the application was going into its third and fourth level of exception, and that's the way it should be.

Performance

Performance is a bigger deal than you might think. I recently trained a group of users on a new web-based system that would on occasion take several minutes for a submenu to appear -- the industry standard is between two and three seconds. The performance of the system broadcasted the quality of the rest of the system, and it wasn't positive.
I know performance issues can be difficult to track down, but that's your problem, not the users. Make sure reasonable response times are documented in your requirements and thoroughly tested when the system is built.

Supportability

Supportability is the last, but not the least important, component of a robust design. Whether you're designing a product that will be used by customers or an internal system that will be used by employees, it's vitally important that the operations group is in a good position to support the solution.
For analytic solutions, supportability often extends beyond requirements into organizational design. The instrumentation on an analytic solution is often sophisticated, so it's important to staff the operations function with very knowledgeable technicians -- maybe even other data scientists. When I'm putting together a development team, I often include at least one person from the support team; this way, they can influence the solution's design from the perspective of someone who's going to support it.

Monday, 15 September 2014

The Google Cloud Platform: 10 things you need to know

The Google Cloud Platform comprises many of Google's top tools for developers. Here are 10 things you might not know about it.


The infrastructure-as-a-service (IaaS) market has exploded in recent years. Google stepped into the fold of IaaS providers, somewhat under the radar. The Google Cloud Platform is a group of cloud computing tools for developers to build and host web applications.

It started with services such as the Google App Engine and quickly evolved to include many other tools and services. While the Google Cloud Platform was initially met with criticism of its lack of support for some key programming languages, it has added new features and support that make it a contender in the space.

Here's what you need to know about the Google Cloud Platform.

1. Pricing

Google recently shifted its pricing model to include sustained-use discounts and per-minute billing. Billings starts with a 10-minute minimum and bills per minute for the following time. Sustained-use discounts begin after a particular instance is used for more than 25% of a month. Users receive a discount for each incremental minute used after they reach the 25% mark. Developers can find more information here.

If you're wondering what it would cost for your organization, try Google's pricing calculator.

2. Cloud Debugger
The Cloud Debugger gives developers the option to assess and debug code in production. Developers can set a watchpoint on a line of code, and any time a server request hits that line of code, they will get all of the variables and parameters of that code. According to Google blog post, there is no overhead to run it and "when a watchpoint is hit very little noticeable performance impact is seen by your users."

3. Cloud Trace
Cloud Trace lets you quickly figure out what is causing a performance bottleneck and fix it. The base value add is that it shows you how much time your product is spending processing certain requests. Users can also get a report that compares performances across releases.

4. Cloud Save

The Cloud Save API was announced at the 2014 Google I/O developers conference by Greg DeMichillie, the director of product management on the Google Cloud Platform. Cloud Save is a feature that lets you "save and retrieve per user information." It also allows cloud-stored data to be synchronized across devices.

5. Hosting
The Cloud Platform offers two hosting options: the App Engine, which is their Platform-as-a-Service and Compute Engine as an Infrastructure-as-a-Service. In the standard App Engine hosting environment, Google manages all of the components outside of your application code.

The Cloud Platform also offers managed VM environments that blend the auto-management of App Engine, with the flexibility of Compute Engine VMs.The managed VM environment also gives users the ability to add third-party frameworks and libraries to their applications.

6. Andromeda
Google Cloud Platform networking tools and services are all based on Andromeda, Google's network virtualization stack. Having access to the full stack allows Google to create end-to-end solutions without compromising functionality based on available insertion points or existing software.

According to a Google blog post, "Andromeda is a Software Defined Networking (SDN)-based substrate for our network virtualization efforts. It is the orchestration point for provisioning, configuring, and managing virtual networks and in-network packet processing."

7. Containers
Containers are especially useful in a PaaS situation because they assist in speeding deployment and scaling apps. For those looking for container management in regards to virtualization on the Cloud Platform, Google offers its open source container scheduler known as Kubernetes. Think of it as a Container-as-a-Service solution, providing management for Docker containers.

8. Big Data
The Google Cloud Platform offers a full big data solution, but there are two unique tools for big data processing and analysis on Google Cloud Platform. First, BigQuery allows users to run SQL-like queries on terabytes of data. Plus, you can load your data in bulk directly from your Google Cloud Storage.

The second tool is Google Cloud Dataflow. Also announced at I/O, Google Cloud Dataflow allows you to create, monitor, and glean insights from a data processing pipeline. It evolved from Google's MapReduce.

9. Maintenance
Google does routine testing and regularly send patches, but it also sets all virtual machines to live migrate away from maintenance as it is being performed.

"Compute Engine automatically migrates your running instance. The migration process will impact guest performance to some degree but your instance remains online throughout the migration process. The exact guest performance impact and duration depend on many factors, but it is expected most applications and workloads will not notice," the Google developer website said.

VMs can also be set to shut down cleanly and reopen away from the maintenance event.

10. Load balancing
In June, Google announced the Cloud Platform HTTP Load Balancing to balance the traffic of multiple compute instances across different geographic regions.

To more about Big Data Hadoop Training in Jaipur please Visit on --

http://www.bigdatahadoop.info/

To More Visit - http://www.techrepublic.com/article/the-google-cloud-platform-10-things-you-need-to-know/

Thursday, 11 September 2014

The Early Release Books Keep Coming: This Time, Hadoop Security

We are thrilled to announce the availability of the early release of Hadoop Security, a new book about security in the Apache Hadoop ecosystem published by O’Reilly Media. The early release contains two chapters on System Architecture and Securing Data Ingest and is available in O’Reilly’s catalog and in Safari Books.

Hadoop security

The goal of the book is to serve the experienced security architect that has been tasked with integrating Hadoop into a larger enterprise security context. System and application administrators also benefit from a thorough treatment of the risks inherent in deploying Hadoop in production and the associated how and why of Hadoop security.

As Hadoop continues to mature and become ever more widely adopted, material must become specialized for the security architects tasked with ensuring new applications meet corporate and regulatory policies. While it is up to operations staff to deploy and maintain the system, they won’t be responsible for determining what policies their systems must adhere to. Hadoop is mature enough that dedicated security professionals need a reference to navigate the complexities of security on such a massive scale. Additionally, security professionals must be able to keep up with the array of activity in the Hadoop security landscape as exemplified by new projects like Apache Sentry (incubating) and cross-project initiatives such as Project Rhino.

Security architects aren’t interested in how to write a MapReduce job or how HDFS splits files into data blocks, they care about where data is going and who will be able to access it. Their focus is on putting into practice the policies and standards necessary to keep their data secure. As more corporations turn to Hadoop to store and process their most valuable data, the risks with a potential breach of those systems increases exponentially. Without a thorough treatment of the subject, organizations will delay deployments or resort to siloed systems that increase capital and operating costs.

The first chapter available is on the System Architecture where Hadoop is deployed. It goes into the different options for deployment: in-house, cloud, and managed. The chapter also covers how major components of the Hadoop stack get laid out physically from both a server perspective and a network perspective. It gives a security architect the necessary background to put the overall security architecture of a Hadoop deployment into context.

The second available chapter is on Securing Data Ingest it covers the basics of Confidentiality, Integrity, and Availability (CIA) and applies them to feeding your cluster with data from external systems. In particular, the two most common data ingest tools, Apache Flume and Apache Sqoop, are evaluated for their support of CIA. The chapter details the motivation for securing your ingest pipeline as well as providing ample information and examples on how to configure these tools for your specific needs. The chapter also puts the security of your Hadoop data ingest flow into the broader context of your enterprise architecture.

We encourage you to take a look and get involved early. Security is a complex topic and it never hurts to get a jump start on it. We’re also eagerly awaiting feedback. We would never have come this far without the help of some extremely kind reviewers. You can also expect more chapters to come in the coming months. We’ll continue to provide summaries on this blog as we release new content so you know what to expect.

If anyone want to learn Big Data Hadoop Training than Visit on - http://www.bigdatahadoop.info