previous article
Architecture recommendation and data-processing techniques with Azure Stream Analytics and Azure Functions. In this article, we provide two architecture recommendations, show how they can be implemented, and visualize the data acquired via IoT Central in a Power BI dashboard. Here you can read part 1.
In addition to data ingestion, data processing in the Industrial Internet of Things (IIoT) is still a major challenge for many companies. How companies successfully implement IoT projects on the basis of a 6-point plan, can be read here. An easy start in connecting industrial devices to the cloud has been described here. Also shown is how IoT Central can be used to read an industrial robot's data from an OPC-UA server, and deposit the data in Azure Blob Storage.
Industrial demand for cloud computing platforms such as Microsoft Azure is growing steadily. The resultant scalability and available IoT stack allow rapid ingestion, processing and analysis of industrial data from sources such as SCADA, as well as connection of different ERP & MES systems.
After having discussed the Azure Synapse service and provided two architecture recommendations for batch as well as (near) real-time processing in the first part of the data-processing series, we discuss serverless processing with Azure Stream Analytics and Azure Functions in the second part.
(Near-) real-time processing with Azure Stream Analytics
In the industrial Internet of Things, there are millions of sensors for which high-performance analyses and (near) real-time insights into data are required. Azure Stream Analytics (ASA) makes it possible to connect different data sources and produce important insights into the data in just a few clicks. With ASA on IoT Edge, analysis models can be deployed close to devices and thus allow reactions to device data in real time. A blog article published by Microsoft describes eight benefits of using Azure Stream Analytics.
In the first step, data provided by IoT Central are routed to ASA via an event hub, transformed using a customized C# function, and then visualized in Power BI.
Azure Blob Storage is used to record the data and enrich analyses performed by ASA with more information. Data provided by the event hub are in parquet format, and transformed in two central steps.
The required robot data are located in the binary column titled "body", and must first be brought into a readable format. A transformation routine in C# is used to perform conversion and obtain a JSON string format.
The columns nested in the string are unpacked using another C# function and written to a Power BI data set. Included in the tutorial Visualisation of fraudulent calls in Power BI: A description of how to connect ASA to Event Hub and set up a Stream Analytics job.
The described procedure of data acquisition via IoT Central is very suitable for getting started with Azure IoT and fast visualization of IoT data. The next step is to connect IoT devices via the Azure IoT hub, and use Azure IoT Edge runtime.
(Near-) real-time processing with Azure Functions
Azure Functions is another way to process IoT data in Azure. In our architecture recommendation, we use Azure Functions to read data from the event hub and then store these data in the Azure Cosmos DB. An Azure Synapse link is used for integration between the Cosmos DB and Azure Synapse Analytics.
This allows real-time reports to be created from the data stored in Azure Cosmos DB, and used for more complex analytics with Azure Synapse. Synapse can be used to store the data in an Azure Data Lake for keeping records and training machine-learning models.
Data visualization with Power BI
In all four presented architectures, data has been enriched with further key performance indicators or KPIs (also refer to IoT data processing part 1: Azure Synapse Analytics). Azure Stream Analytics can be used to calculate additional KPIs in further transformation queries via C# or SQL functions. Azure Functions can be used to integrate the transformations into the code, or subsequently implement them in combination with Azure Synapse Analytics. In our use case, the throughput, downtime and downtime rate have also been calculated and then visualized in the Power BI dashboard.
Also displayed are the target and actual cycle times, as well as the target and current actual number of units over time. Indicated finally are various types of error as well as energy and air consumption during the current shift.
In addition, there are further parameters for measuring the performance of industrial facilities, such as overall equipment efficiency (OEE) for production plants, and mean time between failures (MTBF). Being among the most important KPIs of the industry, these are not to be overlooked by this article. Due to the simulated data, however, these could not be calculated realistically, and were thus not included in our presented IoT end-to-end solution.
Do you intend to bring your IoT data to the cloud, or start with visualizations, reports and predictive maintenance in the next step?
We will support you during every step of your journey into the Internet of Things!
