Adult companies think today about how to increase the value of data stored in their data warehouses (DWH) and data lakes (DL), but to cut the costs for maintenance and support. New companies, at the same time, face big doubts on what should they plan to deploy first to get maximum advantage from their data — DWH or DL.

Many people, who look into the future, would say that nowadays is the best time to combine both approaches and to get a double benefit. But let’s firstly briefly look at two already known concepts.

Image for post
Image for post
Photo by janer zhang on Unsplash

According to Wikipedia, data warehouses are systems used for reporting and data analysis. DWHs are invented in the late ’80s and fed by data that came from different other systems, such are accounting or controlling systems. They are central repositories of data for company reporting, they store both current and historical data in one single place which would highly likely be a relational database management system (RDBMS). Single place simplifies and unifies connections to your data. The data warehouse concept implies extraction, transformation, and loading processes (ETL). Generally, it means that data coming to the system will be transformed when loaded. ETL approach helps your company to create a single version of the truth. …


Image for post
Image for post
Photo by Jozsef Hocza on Unsplash

Many programmers use public code repositories like GitHub.com, to host their code. Moreover, it is great to have your code published on GitHub when you are creating a portfolio for your future employers. But what to do with passwords from your projects? How to work on a project with your small but inspired team and easily but safely share credentials like database accounts, API tokens? Saving passwords inside of your code is a way to lose your money. But enterprise-level key-pair managers, e.g. AWS Secrets Manager, are too complicated and expensive for small teams, right?

Well, the answer is no. Using AWS Secrets Manager is not harder than buying a coffee for your office and sometimes cheaper than it! You will know how to use it easily in your project and keep your sensitive information safe after spending just 5 minutes on this article. …


Earlier, we developed a process of creating a SOAP web service for interacting with SAP Business Warehouse (BW) ETL processes (aka process chains). Web services could really help integrate functionality across the systems in your company’s landscape. Moreover, we can benefit from integration between web services and some front end tools, let’s say, Telegram messenger.

Image for post
Image for post
Telegram Botfather Logo

Telegram is a simple and free messaging app. You can use Telegram on all your devices at the same time — your messages sync seamlessly across any number of your phones, tablets, or computers. …


SAP ABAP is a high-level programming language created by the German software company SAP SE. ABAP is widely used for programming in a variety of systems based on SAP NetWeaver Application Server including SAP Business Warehouse (BW). SAP BW is software for corporate data warehouses (DWH), it is one of the most popular solutions across Europe.

Image for post
Image for post
SAP Business Warehouse Logo

A significant part of SAP BW supports daily work time is usually spend in planning, starting, and analyzing process chains. Process chains are used for organizing data manipulation processes like extraction, transformation, and loading (ETL). …


How often did you dream about a native iOS tool that provides an ability to visualize essential data of your company in a simple and rapid way?

If you ask me, the answer is — every day before I had found the Numerics app from Cynapse. But first things first.

Let’s talk about the real tasks. I have had a request from a client for creating a new dashboard with data gathered from open API. It must provide information about prices of Top 8 world tradable currencies, prices must be shown in Russian Rubles. Source data hosted here: https://exchangeratesapi.io. Our client is going to monitor these indicators on different iOS devices (watches, TVs, and iPads). They do not want to buy new hardware or rent a VPC for such a small project. …


We used to deploy our software on physical or virtual servers with pre-installed environments. It is easy to copy your Python script to that machine and run it via the command line. But what if you need this operation to be done once a month within a specific environment. Do we need to prepare a whole computing instance for this small operation? Another example. We need our code executed each time it triggered, but we cannot predict how often it will happen.

Thanks to AWS we have a simple solution. There is a service called AWS Lambda. Based on the information provided on the official page, AWS Lambda lets you run code without provisioning or managing servers — all with zero administration. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app. Let’s see how to set AWS Lambda function to execute Python script from the internet call. …


Image for post
Image for post
Photo by Kyle Sung on Unsplash

Amazon is a widely-known leader in many areas, especially famous for its flexible superfast cloud web services. Amazon Web Services (AWS) offers more than 200 services for different purposes, such as computing nodes, database instances, file storage, and so on.

The purpose of this article is to provide you with easy step-by-step information on how to set up AWS Relational Database Service for your studies or for your new work project. It is assumed that you already have a working AWS billing account.

  1. Set up a new VPC or use one that you already have. VPC stands for Virtual Private Cloud. It is a mechanism that helps split resources in your AWS account like they are in the different network segments. Do not forget to use 2 different CIDR’s. …

Let’s decompose a simple query and see what’s it consists of:

Image for post
Image for post
SQL Query Example

This particular query consist of 7 elements we could distinguish:

SELECT — keyword that answers the question “what we are going to do with data?”.
currency — here we list column names that we are going to select, we must separate them by commas. It is possible to retrieve all columns, in this way asterisk “*” must be put here instead of column names.
FROM — obligatory keyword pointing to the data array we will look into.
cct — name of the database table that we are querying. It is not necessary to use a database table here, it could be another SQL statement as well.
WHERE — non-obligatory keyword marks the start of conditions. We have to use it almost every time for better performance. It looks like a filter in most cases, in spite of this it is not a filter. We could put any statement after this keyword and the SQL engine will define is it true or false for each returned row.
country = ‘Italy’ — is a part of the condition statement. Here it is a column country we will use for filtering source data, equation operator =, and a string constant ‘Italy’. If we need several conditions in one query we have to use AND/OR after each. …


SQL stands for Structured Query Language. It is actually a programming language that we use for querying databases. Different RDBMS (relational database management systems) support different specifications, but the main query structure, keywords, and operators would be the same among all SQL software.

I use SQL on a daily basis but it does not mean you will do. So, do you actually need to know SQL? When I thought about this I made the following two-part conclusion:

Firstly, the most valuable thing about SQL knowledge is that it helps to construct formal sentences. …


Good day everyone. I’ve started this blog because I suddenly realized its necessity. One day it became clear to me that I need a place where I could categorize and structure all the things I know about my work. Moreover, it could be a good place to write about new tools and technologies I’m learning.

As you can see in my profile, my daily work is tightly related to data gathering, manipulation, and visualization. …

About

Aleksandr Gordienko

Data Engineer | Helping you increase the value of business data

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store