What is BIG DATA

-

 As Name suggests BIG Data is huge data created due to IOT, Apps and real time application data colelction

• Walmart handles more than 1 million customer transactions every hour.

• Facebook stores, accesses, and analyzes 30+ Petabytes of user generated data.

• 230+ millions of tweets are created every day.

• More than 5 billion people are calling, texting, tweeting and browsing on mobile phones worldwide.


The three different formats of big data are:

1. Structured: Organised data format with a fixed schema. Ex: RDBMS

2. Semi-Structured: Partially organised data which does not have a fixed format. Ex: XML, JSON

3. Unstructured: Unorganised data with an unknown schema. Ex: Audio, video files etc.

The core problem which it handles is -

Core is 3V-

  • Volume
  • Velocity
  • Variety 



It helps us on Analytics of prediction & perspective 


Hadoop implementation

  • Amazon EMR
  • MapR
  • Cloudera + Hortonworks - cloudera manager (CDH earlier and now CDP )
  • Azure BigInsights


Apache Oozie - used for workflow for Hadoop tack stacks 

Comments

Post a Comment

Popular posts from this blog

Spark Cluster

-
Image
Spark applications run as independent sets of processes on a cluster, coordinated by the   SparkContext   object in your main program (called the   driver program ). Specifically, to run on a cluster, the SparkContext can connect to several types of  cluster managers  (either Spark’s own standalone cluster manager, Mesos or YARN), which allocate resources across applications. Once connected, Spark acquires  executors  on nodes in the cluster, which are processes that run computations and store data for your application. Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors. Finally, SparkContext sends  tasks  to the executors to run. There are several useful things to note about this architecture: Each application gets its own executor processes, which stay up for the duration of the whole application and run tasks in multiple threads. This has the benefit of isolating applications from ...
Read more

DORA Metrics

-
 DORA metrics, often referred to as the DORA (DevOps Research and Assessment) metrics, are a set of key performance indicators (KPIs) used to measure and assess the effectiveness of an organization's DevOps practices. DORA metrics were developed by the DevOps Research and Assessment team, which includes notable figures in the DevOps community such as Dr. Nicole Forsgren and Jez Humble. The DORA metrics encompass four key areas: 1. **Deployment Frequency:** This metric measures how often an organization deploys code changes to production. Higher deployment frequencies indicate a more agile and responsive development process. 2. **Lead Time for Changes:** This metric tracks the time it takes for code changes to go from code commit to being deployed in production. Shorter lead times suggest efficient and streamlined development and delivery processes. 3. **Mean Time to Recover (MTTR):** MTTR measures how quickly an organization can recover from failures or incidents in production. Low...
Read more