Blog

New data sources and areas of application emerging constantly continue to drive the steady expansion of data storage systems such as DWH, Data Lake and Analytics Platform. Data management processes must also keep pace with growing requirements. Not seldom do small BI applications grow into major initiatives in which several development teams participate. The situation is exacerbated in many industries by the need to make adjustments faster than ever before. This demands short reaction times and high flexibility from the teams and, not least of all, from the infrastructure.

The Same Problem: Listing Folders and Drives

In the last blog, we used a recursive function for a solution with less than 10 lines to scan folders and allow file evaluation by modification date and size.

Now I’m going to raise the bar somewhat for this example by showing even better alternatives.

The problem: listing folders and drives

Recently while working on a project, a colleague asked whether one could list the content of drives in Python. Of course, you can. Moreover, since this isn’t at all complicated, I’d like to take this case to illustrate key best practices recommended for working with paths on drives.

What Can Snowflake Do as a Cloud DB?

Because (storage) volume and execution time are paid for, the shorter times can reduce costs. A detailed online documentation is available at the following URL: https://docs.snowflake.net/manuals/index.html

Incidentally, one does not have to be an AWS customer to be able to use Snowflake. As a cloud-DB service, Snowflake itself offers no proprietary ETL tools, but leaves this to the manufacturers of ETL, reporting or self-service BI tools. These usually provide native drivers and connections to allow use of their tools with Snowflake. If no separate ETL tool is to be used at the company, there are several possibilities of loading data and realizing the ETL routes. One possibility is implementation of logic in SQL, and orchestration via Python.

What to consider before starting the Development

Recommendation systems are a crucial part of every digital business model. This blog post concisely answers two foundational questions:

1. Who should care about recommendation systems and why?
2. What are the primary flavors of recommendation systems? How much work is it to implement them?

In this article I focus on a solid overview.

A lot of what have hitherto been manual programming tasks can be replaced - or at least greatly simplified - by DWA tools. The precise elements of development that can be automated in that respect can vary greatly from tool to tool. For example there are pure code generator concepts, with the aid of which database structures and ETL/ELT processes can be generated automatically ("design time"). On the other hand, extensive integration suites exist that can generate but also manage the entire DWH lifecycle, from provision of the data in the sources right through to the data marts ("run time").

During the development phase there is a series of tasks in which a DWA tool can provide support. The following deals in particular with the fields of reverse engineering and compatibility, analysis, implementation and framework condition.

Increased use of the Internet as well as smartphones is also increasingly changing consumer behavior. With increasing digitization of channels and touchpoints, customers today are informed better than ever and thus harder to acquire. To register, evaluate and understand consumer behavior digitally too, persons in charge of marketing must today form targeted segments and create an accordingly coordinated strategy for potential customer touchpoints, in order to address customers on the right channel at the right time with the right offer, while nonetheless refraining from inundating them with e-mails or push-messages.

Particularly in the environment of HCM and Infotype tables, full loads are often used for supplying SAP BW because clear detection of changes (CRUD - create, read, update and delete) in the source system is not possible. For example, deadlines (Infotype 0019) are not furnished with a full validity, so that a deletion does not lead to creation of a new data record which overwrites this period. In the source system, SLT/RS (SAP landscape transformation/replication server) with its trigger-based detection of changes at the database level could be used to monitor each individual data modification. Due to the relatively small amounts of data, however, use is not absolutely necessary, and change-tracking at this point would be "overkill". It should be noted that SLT/RS is sometimes included in current SAP BW license packages, and can therefore often be used, but requires an installation on the source system.

In the first part of this blog, the aim was to serialize a CSV file as simply as possible to Avro, and store the result in Kafka, the schema being registered in the related registry.

An important building block for a planning system comprises dealing with intercompany (IC) issues. IC comes into play whenever different economic entities of a group of enterprises exchange goods or services with each other. The resulting expenditures and earnings, as well as receivables and payables must be transparent and definable with respect to each other. Because planning is performed by numerous participants, IC coordination and subsequent IC elimination are not trivial.

Even in modern environments, CSV is still a frequently encountered exchange format because many existing systems cannot deal with more modern alternatives. However, other formats are better suited to further processing in a big-data environment. This applies, in particular, to Avro in conjunction with Kafka. Avro offers a space-saving data format with many features, in which the data schema is also transferred. To improve handling, the schema can also be registered in a related repository.

Interactions with customers in the field of energy supply are rare – a single contact a year is not uncommon. Best possible contact management is correspondingly important. This project focuses on the communication channel comprising the inbound call center.