HADOOP \ SPARK ECO System

-

The Big data eco system is build with multiple tools and systems.

1) What is difference between sqoop and flume?

Sqoop and Flume both are meant to fulfill data ingestion needs but they serve different purposes. Apache Flume works well for streaming data sources that are generated continuously in hadoop environment such as log files from multiple servers whereas whereas Apache Sqoop works well with any RDBMS has JDBC connectivity.


2) What is Hive?

  • Hive is a data warehouse infrastructure tool that processes structured data in Hadoop. It is proced by FB to help data anlytics \ RDBMS people to directly query Hadoop cluster.
  • Most data warehousing applications work with SQL-based querying language.
  • Features of Hive. It accelerates queries as it provides indexes, including bitmap indexes.




Comments

Post a Comment

Popular posts from this blog

Spark Cluster

-
Image
Spark applications run as independent sets of processes on a cluster, coordinated by the   SparkContext   object in your main program (called the   driver program ). Specifically, to run on a cluster, the SparkContext can connect to several types of  cluster managers  (either Spark’s own standalone cluster manager, Mesos or YARN), which allocate resources across applications. Once connected, Spark acquires  executors  on nodes in the cluster, which are processes that run computations and store data for your application. Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors. Finally, SparkContext sends  tasks  to the executors to run. There are several useful things to note about this architecture: Each application gets its own executor processes, which stay up for the duration of the whole application and run tasks in multiple threads. This has the benefit of isolating applications from ...
Read more

DORA Metrics

-
 DORA metrics, often referred to as the DORA (DevOps Research and Assessment) metrics, are a set of key performance indicators (KPIs) used to measure and assess the effectiveness of an organization's DevOps practices. DORA metrics were developed by the DevOps Research and Assessment team, which includes notable figures in the DevOps community such as Dr. Nicole Forsgren and Jez Humble. The DORA metrics encompass four key areas: 1. **Deployment Frequency:** This metric measures how often an organization deploys code changes to production. Higher deployment frequencies indicate a more agile and responsive development process. 2. **Lead Time for Changes:** This metric tracks the time it takes for code changes to go from code commit to being deployed in production. Shorter lead times suggest efficient and streamlined development and delivery processes. 3. **Mean Time to Recover (MTTR):** MTTR measures how quickly an organization can recover from failures or incidents in production. Low...
Read more