Bosch IoT Insights

Data processing

Bosch IoT Insights is the central place to collect data from different sources. Before device data can be examined, combined, and visualized, it needs to be processed. Raw device data is often encoded in a non-human-readable technical format. For example, hexadecimal byte strings or complex XML structures. Another challenge are device versions and variants that may send data in incompatible structures. Bosch IoT Insights addresses these project complications with data processing. Data processing allows projects to transform, normalize, enrich, and clean the incoming data.

images/confluence/download/attachments/1298258636/image2020-2-20_16-57-52.png

Features

  • Decoding
    Turns binary data into human-readable data. The decoding feature supports various industry-wide description standards, such as ODX, Fibex, A2L, MDF, DBC, and Google Protocol buffer.

  • Normalization
    Unifies data by bringing together different data sources and unifying their data structures.

  • Duplicate removal
    Removes duplicates to guarantee high data quality with reduced costs.

  • Filtering
    Specific queries can be defined to filter data.

  • Aggregation
    Different data sources can be aggregated together.

  • Forwarding
    Data can be forwarded via the REST API to other cloud services.

  • Alarm
    Specific alarm messages or triggers can be configured and sent via email, SMS, and webhooks.

  • Custom code
    Custom processing scripts (Java, JavaScript, and Python) can be combined with other processors. For an example on how to configure the custom step, refer to Pipelines: Configuring the custom step.

images/confluence/download/attachments/1298258636/image2019-12-19_10-33-4.png

Components

In Bosch IoT Insights, the Data Recorder acts as an endpoint for sent device data and stores the data in a raw format. The Processor component improves the raw data.

images/confluence/download/attachments/1298258636/image2020-2-20_13-46-16.png

A processor contains defined improvement steps. The raw data is processed by those improvement steps until its final target format is reached. This chain of improvement steps is called Pipeline. In a Bosch IoT Insights project, pipelines can be configured with an editor. The editor provides several improvement/processing steps. The editor also allows you to upload your own project-specific processing logic.

File formats

The following file formats are supported:

File format

File extension

Description

ASAM MDF

mdf

mf4

Binary file format that stores recorded or calculated data, refer to ASAM MDF

Comma-Separated Values

csv

Delimited file format that separates values with a comma

Tab-Separated Values

tsv

File format that stores values in a table

Vector DBC1

dbc

File format that describes data over a CAN bus

ASAM ODX1

odx

pdx

File format for vehicle diagnostics, refer to ASAM MCD-2 D

ASAM-A2L1

a2l

File format for internal ECU variables, refer to ASAM MCD-2 MC

ASAM-Fibex1

dcm

File format for ECU network systems, refer to ASAM MCD-2 NET

Extensible Markup Language

xml

File format for storing semi-structured data

Protocol Buffers1

proto

File format developed by Google to serialize data, refer to Protocol Buffers

GPS Exchange Format

gpx

File format to store location data

iTRAMS

bin

Bosch-specific binary file format that extends the Vector DBC file format

1 The actual input for this file format are hex-encoded binary files, containing measurement information. The mentioned file format contains the meta information for decoding the given binary file.

The standard processing of Bosch IoT Insights also supports zipped content (zip, gzip, and inflate) to save storage space.