Birla Institute of Technology & Science, Pilani
Work Integrated Learning Programmes Division
Second Semester 2020-2021
Comprehensive Examination
(EC-3 Make-up)
Course No. : DSECL ZG556
Course Title : STREAM PROCESSING AND ANALYTICS
Nature of Exam : Open Book
Weightage : 45%
Duration : 2 Hours
Date of Exam : 11-09-2021 FN
Note to Students:
Please follow all the Instructions to Candidates given on the cover page of the answer book.
All parts of a question should be answered consecutively. Each answer should start from a fresh page.
Assumptions made if any, should be stated clearly at the beginning of your answer.
Q1. Every day a multinational online taxi dispatch company gathers terabytes of event data from its mobile users. By using Kafka, Spark Streaming, and HDFS, to build a continuous ETL pipeline, they can convert raw unstructured event data into structured data as it is collected, and then use it for further and more complex analytics. [5 + 5 = 10]
With this scenario in mind, explain how Spark Streaming will be leveraged as solution using a nicely labelled architecture diagram?
List and briefly explain the Apache Spark API's that can be used in?
Q.2. Consider the following Kafka Cluster description.
10 node cluster
Name of the Topic: Cluster
Number of Partitions: 4
The Replication factor of ‘Bus: 3
7 producers
5 consumers
Draw Kafka’s architecture clearly highlighting the following in a block diagram
-producers, consumers, broker, topic and partitions.
How many consumer groups can be created for this configuration?
What is the maximum number of consumers that each consumer group can have while ensuring maximum parallelism?
What is the maximum number of server failures that this setup can handle?
[2 + 1 + 1 + 1 = 5]
Q.3. Explain the various components available in the Apache Flink with suitable real time example.
[10]
Q4. Consider an online ecommerce portal where customers can search for the products anonymously but for placing the order, they need to have the account with the provider. When customers are browsing the products on the portal, their online behavior is getting monitored by the provider. The provider has business relationship with another online movie service provider whose movies are also displayed and sold on the provider’s platform. Also the users search queries are shared between these providers. The search queries are also matched with the users profile to provide product / movie recommendations to the users. For this purpose it makes use of Apache Storm as streaming platform. With the help of suitable architectural diagram, represent how this recommendation activity can be carried out. [8]
Q5. Consider the following streaming SQL query where an output record (or row) is generated specifying the updates to the minimum and maximum temperatures over the window W1, plus an incrementally updated average for the temperature over that period. [3 * 4 = 12]
SELECT STREAM
MIN(TEMP) OVER W1 AS WMIN_TEMP,
MAX(TEMP) OVER W1 AS WMAX_TEMP,
AVG(TEMP) OVER W1 AS WAVG_TEMP
FROM WEATHERSTREAM
WINDOW W1
Let’s assume that input streaming weather stream has following temperature values coming in at regular interval of two minutes:
{12, 14, 15, 13, 16, 20}
What will be the output of the above query (with proper explanation) if
Window is defined as sliding window of length 3
Window is defined as batch window of length 3
Window is defined as sliding window of time 4 minutes
Window is defined as batch window of time 3 minutes
