The Data Engineer Blog

Posted on December 9, 2020

As in the first release (1.0.0), SchemaMapper was developed to merge data from different file types (flat file, Excel, Access …) into one SQL table. SchemaMapper 1.1.0 was released after being improved to support reading from relational databases and writing data into more data formats. Also, SchemaMapper 1.1.0 is now... [Read More]

Posted on December 7, 2020

This article will briefly mention nine features added starting SQL Server 2008 that make SQL Server more than a traditional database management system.

Posted on October 14, 2020

In this article, we will explain how to build databases using BIML. We will be covering handling databases objects such as schemas, tables, columns, indexes, and other objects.

Posted on September 14, 2020

In this article, we will be talking about Manticore Search, which is an open-source search engine first released in 2017 as a fork of the Sphinx search engine. We will try to describe this search engine briefly, mention some of its differences from the Sphinx search engine, and we will... [Read More]

Posted on August 4, 2020

In this article, we will talk about building full-text indexes using Sphinx. We will be covering seven topics: The Sphinx configuration file Defining data sources Defining full-text indexes Indexer and Searchd services options Database preparation Indexing Using Linked servers to access indexes

Posted on July 11, 2020

In this article, we will be talking about the Sphinx search engine and how to use it to install it on the Windows operating system. This is the first article in the “Sphinx Search Engine” series, where we will explain how to install and use this search engine to create... [Read More]

Posted on May 19, 2020

In this article, we will talk briefly about data quality in SQL Server. Then, we will give a brief overview of the Melissa Data Quality for SQL Server Integration Services (SSIS), and we will demonstrate the components available in the community edition.

Posted on May 15, 2020

After working on several projects, I ended up with a consequence that it is not recommended to install Hadoop on windows operating system. In this article, I briefly described the main reasons for this consequence.

Posted on May 14, 2020

In this article, we will be talking about Hadoop Hive and Hadoop Pig Tasks. We will first give a brief overview of Apache Hive and Apache Pig. Then, we will illustrate the related SSIS Hadoop components and alternatives.

Posted on May 11, 2020

In this article, we will give a brief introduction of Hadoop and how it is integrated with SQL Server. Then, we will illustrate how to connect to the Hadoop cluster on-premises using the SSIS Hadoop connection manager and the related tasks.

Posted on May 11, 2020

In this article, we will briefly explain the Avro and ORC Big Data file formats. Then, we will be talking about Hadoop data flow task components and how to use them to import and export data into the Hadoop cluster. Then we will compare those Hadoop components with the Hadoop... [Read More]

Posted on May 5, 2020

This article is a part of a series that we are publishing on TowardsDataScience.com that aims to illustrate how to install Big Data technologies on Windows operating system.

Posted on May 4, 2020

While working on a project two years ago, I wrote a step-by-step guide to install Hadoop 3.1.0 on Ubuntu 16.04 operating system. Since we are currently working on a new project where we need to install a Hadoop cluster on Windows 10, I decided to write a guide for this... [Read More]

Posted on April 22, 2020

In this article, we will explain what JSON is, what are the SQL Server JSON functions introduced in 2016, how to index JSON values, and how to import JSON data into SQL tables.

Posted on April 18, 2020

While working on a project, we were asked to install Apache Hive on a Windows 10 operating system. Many guides were found online but unfortunately, they didn’t work. For this reason, I decided to write a step-by-step guide to help others.

Posted on April 14, 2020

Glad to annouce that we have created a NuGet package for the SchemaMapper class library. You can check it on the following link: NuGet Gallery - SchemaMapper 1.1.0

Posted on April 2, 2020

DTExec is a tool used to configure and execute SQL Server Integration Services (SSIS) packages. Many developers and database administrators use this tool, especially when automating the package execution using third-party applications or non-supported programming languages. This article will introduce this tool to you, also help you understand how and... [Read More]

Posted on March 26, 2020

In this article, we will illustrate another Biml alternative, which is the EzApi class library, and we will make a comparison between both technologies.

Posted on March 25, 2020

In this article, we will first illustrate how to create, save and execute SSIS packages using ManagedDTS in C#, then we will do a small comparison with Biml.

Posted on March 18, 2020

In this article, we will explain the File Transfer Protocol (FTP), and we will give an overview of the FTP task in SQL Server Integration Services (SSIS) and FTP connection manager.

Posted on March 18, 2020

In this article, we will explain how to store these VB or C# scripts within external files and use them within Biml scripts. Also, we will mention how to create extension methods within these C# scripts.

Posted on March 16, 2020

In this article, we will talk about sp_WhoIsActive stored procedure and how we can use it to monitor currently running activities in SQL Server.

Posted on March 16, 2020

In this article, we will explain BimlScript, which is an extension of this markup language with VB or C# scripts.

Posted on March 13, 2020

In this article, we will explain how to generate scripts from existing SSIS packages by showing all related options. Then, we will analyze the generated script and identify how each object is mentioned in the script.

Posted on March 13, 2020

In this article, we will explain how to generate SSIS packages using business intelligence markup language by providing a step-by-step guide where we will illustrate how to configure each object and how to use variables, expressions and parameters within the package.