Birla Institute of
Technology & Science, Pilani
Work Integrated
Learning Programmes Division
First Semester 2022-2023
Mid-Semester Test
(EC-2 Makeup – ANSWER KEY)
Course No. :
DSECL ZC556
Course Title :
Stream Processing and Analytics
Nature of Exam :
Open Book
No. of Pages = 4
No. of Questions = 5
|
Weightage : 30%
Duration :
2 Hours
Date of Exam : 06/03/2021 or 19/03/2021 (FN/AN)
Note to Students:
1. Please follow all the Instructions
to Candidates given on the cover page of the answer book.
2. All parts of a question
should be answered consecutively. Each answer should start from a fresh
page.
3. Assumptions made if any,
should be stated clearly at the beginning of your answer.
Q1. Consider an online food
ordering and delivery platform which enables the customers to browse the nearby
serving restaurants any time, explore their menu options, order the food and
get it delivered at their doorsteps, also provide the ratings for the foods.
The platform also enables the restaurant owners to analyze the click-stream as
well as historical data related to their restaurant so that they can improvise
their decisions with respect to the promotional offers made on the platform. [1 +1 + 2 + 1 + 1 = 6]
a)
Identify
the different data sources involved in this scenario. Also label them as
internal or external.
b)
If
you have been asked to design the customer data stream in this scenario, how it
will look like?
c)
Give
two examples of exploratory data analysis that restaurant owners can perform
with this data? Give details.
d)
Recommend
any suitable machine learning (with adequate justification) that can help the
restaurant owners to improvise their decisions about marketing efforts.
e)
Which
type of system architecture will be useful in this scenario?
Answer:
a) Customer
profile data, Restaurant data, Orders database, Food delivery agents data
All of them are of type internal
to the system
b) It will
be mix of few customer + restaurant + order data
{“timestamp”, “cust_id”,
“rest_id”, “order_id”, “items”, “payment_mode”…..}
c) Some
examples -
Find out factors affecting the
ratings of the restaurant
Find out the relationship
between the items those are ordered together
d) Few
examples –
Clustering – find out customers
who are similar in terms of order, location etc
Regression – to predict the
customer rating for this order or restaurant or to determine when he will place
his next order or estimate the next order pricing
Classification – to predict
whether customer will place an order in next n days or not or identify the
customer to make an offer / discount
Recommendation – recommend the
menu items to the customers
e) Lambda –
historical data for prediction, real time data for ordering and tracking etc.
Q2.
Imagine that you are building a real time traffic routing system. Identify and
justify the communication mechanisms used while making data available to the
external world in the following use cases.
[6]
a)
People
driving around any city can use mobile application to get updates and be
re-routed based on up-to-the moment traffic conditions. The user will post the
interest for a particular route through mobile application and in turn the
system will send the updates about the traffic conditions on that route.
b)
Police
department wants to host web application that has a real time dashboard which
continuously shows the traffic conditions at several points in the city and
take appropriate action to resolve traffic congestions based on the data feeds
those are coming from your streaming systems.
c)
A
device is fitted into the commercial passenger vehicles which are roaming
around the city. This device has the capability to show the current traffic
conditions on the display and send the current coordinates of the vehicle and
other relevant information back to the streaming system.
Answer:
a) Webhooks
User
registers the interest and application sends back the data to the mobile
application
b) Server
sent events (SSE)
Connection
between dashboard and system will remain open and data is made available to the
client to update the dashboard
c) WebSockets
Enables
the two way communication between the device and streaming system
For each
sub-question, Identification of each technique - 1 mark and adequate
justification 1 mark
Q3. Consider a banking systems
that has 4000 ATMs situated across many places in a particular region. The bank
customer can carry out many transactions like balance check, money withdrawal
and deposit, password reset. These transactions takes 4, 8 and 6 bytes
respectively. Each of this transaction has a timestamp attached to it which
takes 8 bytes. This transaction data needs to be stored at different places
like at ATMs, at an immediate buffer at streaming system side and in a
persistent storage. Narrate the three use-cases which require the data to be
stored at these three places and estimate the sizing required for the same and
suitable option for the storage.
[6]
Answer:
a)
At ATM –
May be just transactions data happened in ATM during last 30 minutes
– if something goes wrong with any transaction like wrong password coming again
and again for same customer, immediately raise an alarm to check that the
person trying out the transaction is really card owner or not, local processing
at ATM needs to be done
In memory databases can be used but sizing will vary from machine to
machine and can be better estimated based on historical transaction data for
that machine
b)
At buffer – immediate trend
analysis wrt money withdrawal patterns and use it for feeding the ATMs with
money most of the times
May be a day’s transactions data needs to be persisted in the buffer
Data flow tiers such as Kafka, Flume can be helpful
For storing processed data, caching systems can be useful
c)
At database – for historical
data analysis , both (a) and (b) will get benefited by this sort of permanent
storage like Databases or data stores
Sizing will depend upon business’s opinion about how recent data
needs to be taken into consideration.
For each correct
identification – 1 mark and explanation 1 mark
Q4. Answer the following
questions in brief:
[2 *3 = 6]
a) How Apache Flume blurs
boundaries between data motion and processing?
Answer:
Interceptors
are where Flume begins to blur line between data motion and processing. Interceptor model allows Flume not only to
modify the metadata headers of an event, but also allows the Event to be
filtered according to those headers.
b) Mention two important ways by
which processed streaming data can be made available to the end users?
Answer:
1) Through dashboards - by sending processed data to the
visualizations placed on dashboards
2) Though alerts/notifications - by sending
important/exceptions updates to the users through alerts or other notification
channels
c)
“Apache Kafka adopts a prescriptive order
for reading and writing operations for a topic”. Justify this statement
mentioning whether it is true or false.
Answer:
True. Apache Kafka is not a queuing system
like ActiveMQ. It does not follow the semantics that messages get processed as
they are arrived. Kafka’s partitioning system does not maintain such structure.
Q5. Consider the following block
diagram of data flow system based on Apache Flume. [1 +4 + 1 =6]
a)
Identify essential components from the
perspective of Flume Agent.
b)
Provide the suitable configuration details
for Apache Flume Agent that matches this data flow scenario.
c)
Which type of data flow is represented
with this block diagram?
Answer:
a)
Three essential components of
Flume Agent
·
Sources – Avro source, Thrift
source and Syslog source for port monitoring data
·
Channels – Memory channel
·
Sink – HDFS sink, ElasticSearch
sink
Identification of sources, channels and sinks – 1 mark
b)
Agent configuration should have mention of three sources,
channel and sinks
myAgent.sources = myAvroSource,
myThriftSource, mySyslogSource
myAgent.channels = myMemoryChannel
myAgent.sinks = myHDFSSink,
myESSink
myAgent.source. myAvroSource.type
= avro
myAgent.source.
myThriftSource.type = thrift
myAgent.source.
mySyslogSource.type = syslogtcp
myAgent.sink. myHDFSSink.type = hdfs
myAgent.sink. myESSink.type = elasticsearch
myAgent.channels.
myMemoryChannel.type =memory
myagent.sources.myAvroSource.channel = myMemoryChannel
myagent.sources. myThriftSource.channel = myMemoryChannel
myagent.sources. mySyslogSource.channel = myMemoryChannel
myagent.sinks. myHDFSSink.channel = myMemoryChannel
myagent. sinks. myESSink.channel = myMemoryChannel
Definition of source, sink and
channel – 0.5 mark
Configuration of source – 1 mark
Configuration of channel – 0.5
mark
Configuration of sink – 1 mark
Mapping between source/sink and
channel – 1 mark
b)
Fan-in flow – from multiple
sources to single channel
********************
All the messages below are just forwarded messages if some one feels hurt about it please add your comments we will remove the post. Host/author is not responsible for these posts.
