According to Wikipedia, BI involves the means, the tools and the methods used to collect, consolidate and produce the data model, and to finally display it. It is designed:

• to help a company taking decisions;
• to cover its whole activity for people in charge of the strategy.

Simply put, BI is the art of transforming raw data into valuable information.

This article begins with an introduction to BI, then describes the main actors of the BI market, and what their products are designed for.

Business Intelligence

Setting up a BI solution means building a decision tool from internal and external data. The goal of BI is to provide the information required to take decisions. The main steps of any BI projects include:

1. Collecting data from all the existing sources and putting it into an ODS. Data is simply extracted from RDBMS, ERP, CRM systems, flat files, structured files (XML, CSV, XLS…) and loaded into a DBMS.
2. Consolidating data into the DWH and as many DM as needed (one or more by line of work, depending on the needs). Data is extracted from the ODS and loaded into the DWH and the DM after some transformations. One also uses OLAP Cubes for projects involving multi-dimensional data.
3. Consuming data using front-ends such as web or mobile applications. These applications are built using specific tools and allow the user to display pre-computed dashboards or to explore the information stored in the DWH, the DM or the Cube.
4. Analyzing data with data-mining tools.

BI uses three kinds of technologies (we won’t talk about data-mining):

• data integration tools (also known as ETL or ELT) to collect and consolidate data;
• databases to store data into the ODS, the DWH, the DM or the cubes;
• reporting tools to create dashboard, scorecards and to let end-users explore the data.

Each technology may be used independently and for different purposes.

The following figure describes the data lifecycle from the sources to the end-user.

Data Integration

Data integration is performed using ETL or ELT tools.

An ETL extracts data from different sources, applies transformations (cleansing, renaming, computing) using its own engine (implemented in general-purpose programming languages, such as Java, C++…) and loads data into the target tables. Informatica PowerCenter, IBM Datastage, Microsoft SQL Server Integration Services, SAP BO Integration and Talend Enterprise Data Integration are ETLs.

Unlike in an ETL, the transformation process in an ELT is performed using the target DMBS engine (SQL language), like Oracle Data Integrator.

In both cases, data integration tools move data from many different sources to a single target that may contain many tables. To choose an ETL/ELT tool, a company may consider the cost of the tools, the possibility to load data from all of its sources’ kinds, the language used (Java, SQL, Perl…), as well as the IDE that comes with it.

Databases

Databases store the result of data integration. They need to be really efficient, especially for reading operations, because many end-users may use dashboards or data exploration at the same time and on a lot of data. In contrast, writing operations are often performed off the working hours.

The most popular databases are relational, like Microsoft SQL Server, IBM DB2, Oracle RDBMS, MySQL or PostgreSQL. The choice between them is done depending on their price (some are free, some other are very expensive) but also by looking at the administration tools, the IDE and the SQL extensions provided by the editor. Although RDBMS are used, data modeling is typically done according to a star schema, which does not fit into the normal form.

Teradata Database and Greenplum are also RDBMS but they are specialized in big volumes and use massive parallel processing. For companies with big volumes, it’s almost mandatory to use such databases to keep good performances.

Finally, in order to optimize reading operations, some databases like SAP Sybase IQ are column-oriented (as opposed to RDBMS that are row-oriented).

Reporting

Last but not least, the reporting tools are very important because they are the only mean by which end-users will interact with the information. The reporting market is more complex because there is a large number of editors that are very different from each other. But at the end, the way they work is very similar:

1. Developers must specify the way the tool accesses data: connection to the database, data modeling (using star schema, dimensions and fact tables).
2. Then end-users explore data themselves or / and use pre-computed dashboards.

The leaders in this market are (more because of acquisitions and the editors’ reputation than on the basis of some quality criteria):

• IBM Cognos
• Oracle BI Enterprise Edition
• SAP Business Object
• Microsoft SQL Server Restitution Services

Some challengers are also very competitive and gaining market share, like Qlikview (first to make in-memory systems popular), Microstrategy (one of the best mobile applications as of today), Tableau Software (massively used in the USA), Bime and Bittle (BI in SaaS).

Again, the choice of a reporting tool depends on its price, but also its speed and the ease of access for end-users. Each company has its own needs and constraints.

Full BI Suites

Some editors provide a data integrator, a database, and a reporting tool: so they have a full BI suite. It is the reason why IBM has bought Cognos and Datastage, Oracle has bought Sybase, Hyperion and Sunopsis, and SAP has bought Business Object. Microsoft has had a different strategy and has built its own suite by developing a data integration and a reporting tool on top of Microsoft SQL Server, but the result is the same: a BI suite.

Besides the big ones, there are three BI suites that deserve to be introduced because they are open-source and less expensive, and offer a real alternative to IBM, Oracle or SAP (although Microsoft BI is cheaper):

• Pentaho
• Jaspersoft (using Talend as data integration tool)
• SpagoBI

Challenges

This article has introduced the market of BI as of today.
But since it is an active market, new concepts are now emerging:

• Mobility. People are now working everywhere: at home, at work, in public transports, and the use of tablets, mobile phones or notebooks is very common in meetings. BI reports must evolve and the dashboards’ layouts have to be compatible with mobiles and tablets.
• Data Quality / MDM.
• In-memory applications. Computers are built with increasing RAM capacity. Accessing data from RAM is much more efficient than from disk. In-memory applications use powerful and specific algorithms to compress and access data. A growing number of big editors are selling appliances to support the in-memory concept: Hana by SAP, Exalytics by Oracle.
• Semi-structured and unstructured data. Most of the time, the company’s knowledge is stored as unstructured data, like e-mails, and it is very difficult to analyze it. This is what SBA are trying to deal with.
• Big data. Companies like Facebook, Google or Yahoo have to deal with increasingly large volumes of data. To be able to store, analyze or even access everything rapidly, they have developed new technologies around clusters, parallel architectures and distributed computing, with solutions like the MapReduce framework used by Apache Hadoop (among others), or new database concepts such as NoSQL (because RDBMS do not meet their needs). This challenge is often called “big data” and might be the biggest of 2012, but we will talk about it in another article.

Going further

• Business intelligence (EN)
• Video: History of Business Intelligence (EN)
• What in-memory BI ‘revolution’ (EN)
• In-memory BI is not the future. It’s the past. (EN)
• Mobile Business Intelligence and Analytics (EN)
• Data Management: The Year Past, The Year Ahead (EN)
• BI 2012: The Future that has Already Happened (EN)
• 10 things you Need to Know About NoSQL Databases (EN)
• Visual Guide to NoSQL Systems (EN)
• NOSQL Patterns (EN)
• Big data terminology and positioning (EN)
• Why Search-Based Applications are Set to Thrive (EN)

Credits

• Illustration of the star schema by SqlPac under the Creative Commons Attribution-ShareAlike 3.0 License

]]> http://rtfblog.com/2012/02/07/an-overview-of-the-business-intelligence-world/feed/ 0 cperrod Key steps of BI: collect, consolidate, consume, analyze. Data lifecycle: integration, storage, presentation to the end-user. Example of a star schema. http://rtfblog.com/2011/09/07/update-or-insert/ http://rtfblog.com/2011/09/07/update-or-insert/#comments Wed, 07 Sep 2011 19:55:36 +0000 Clément Perrod http://rtfblog.wordpress.com/?p=422 ]]> Some operations are not as easy in SQL as in procedural or imperative languages. For example while it is very intuitive to make an update or insert (also known as “upsert”) operation on a std::vector in C++, it is tricky in SQL. The upsert operation is: update data if already in the set and insert it otherwise.

As for recursion in SQL, the way to perform an upsert depends on the DBMS and its version.

This article introduces different methods to upsert data in a table. Many solutions are introduced because not all DBMS are compatible with SQL:2008 and the MERGE statement. As of today, it is compatible from these versions:

• Microsoft SQL Server 2008
• IBM DB2 v9
• Oracle 9i

MySQL does not provide this statement but its own SQL extentions:

• REPLACE statement
• ON DUPLICATE KEY clause for INSERT statement

Microsoft SQL Server 2000 and 2005 don't provide this feature but upsert is possible using the output clause for the UPDATE statement.

Finally, the only solution I am aware of for PostgreSQL is to use a cursor and test each line: rows existing in the target table are updated and the others are inserted.

Beware: all examples were written under Microsoft SQL Server 2008 (except the MySQL part). They may use some specific code but should remain fully understandable by everyone used to SQL.

Dataset

Before introducing all methods, let's set up the sample data.

@tUsers is the working table. It contains a list of users with one particular field: active. When inserting data, this field is not set.

declare	@tUsers		table
(
	id		int		NOT NULL,
	firstName	varchar (30)	NOT NULL,
	lastName	varchar (30)	NOT NULL,
	active		tinyint		NULL
)

insert into @tUsers	(id, firstname,	lastname)
values	(1, 'Clément', 'Perrod')

insert into @tUsers	(id, firstname,	lastname)
values	(2, 'Thomas', 'Dupont')

insert into @tUsers	(id, firstname,	lastname)
values	(3, 'John', 'Doe')

The table contains the following rows:

id	firstName	lastName	active
1	Clément	        Perrod	        NULL
2	Thomas	        Dupont	        NULL
3	John	        Doe	        NULL

@tStaging will be the source table for upsert in @tUsers. Some rows are new and others are just an update for the active field.

declare	@tStaging	table
(
	id		int		NOT NULL,
	firstName	varchar (30)	NOT NULL,
	lastName	varchar (30)	NOT NULL,
	active		tinyint		NULL
)

insert into @tStaging	(id, firstname,	lastname, active)
values	(1, 'Clément', 'Perrod', 1)

insert into @tStaging	(id, firstname,	lastname, active)
values	(3, 'John', 'Doe', 0)

insert into @tStaging	(id, firstname,	lastname)
values	(4, 'Elise', 'Duffin')

insert into @tStaging	(id, firstname,	lastname)
values	(5, 'Anonie', 'Mousse')

The table contains following rows:

id	firstName	lastName	active
1	Clément		Perrod		1
3	John		Doe		0
4	Elise		Duffin		NULL
5	Anonie		Mousse		NULL

Methods

MERGE statement

The easiest way to perform upsert is to use the MERGE statement introduced by SQL:2008. Its syntax is pretty intuitive: different treatments are applied on the target table using data from the source table depending on a merging clause. In this article it is used for upsert, but it is also possible to use it to delete data in target table.

MERGE 
    [ TOP ( expression ) [ PERCENT ] ] 
    [ INTO ] <target_table> [ WITH ( <merge_hint> ) ] [ [ AS ] table_alias ]
    USING <table_source> 
    ON <merge_search_condition>
    [ WHEN MATCHED [ AND <clause_search_condition> ]
        THEN <merge_matched> ] [ ...n ]
    [ WHEN NOT MATCHED [ BY TARGET ] [ AND <clause_search_condition> ]
        THEN <merge_not_matched> ]
    [ WHEN NOT MATCHED BY SOURCE [ AND <clause_search_condition> ]
        THEN <merge_matched> ] [ ...n ]
    [ <output_clause> ]
    [ OPTION ( <query_hint> [ ,...n ] ) ]    
;

The upsert is as easy as this:

merge	@tUsers		as us
using	@tStaging	as st
on	us.id = st.id
/*	Update	*/
when matched then
	update
	set		us.active = st.active
/*	Insert	*/
when not matched then
	insert (id, firstName, lastName, active)
	values (st.id, st.firstName, st.lastName, st.active)
/*	Mandatory semi colonn	*/
;

ON DUPLICATE KEY clause of the INSERT statement

This upsert method is based on the ON DUPLICATE KEY clause for the INSERT statement, which is available in MySQL 5 and above.

This solution only works if the table @tUsers has a primary key. Thus, for this example to work, the table declaration is:

CREATE TABLE `tUsers` (
  `id` int(11) NOT NULL,
  `firstName` varchar(30) NOT NULL,
  `lastName` varchar(30) NOT NULL,
  `active` tinyint(4) DEFAULT NULL,
  PRIMARY KEY (`id`,`firstName`,`lastName`)
)
;
INSERT INTO tUsers	(id, firstname,	lastname)
VALUES	(1, 'Clément', 'Perrod')
;
INSERT INTO tUsers	(id, firstname,	lastname)
VALUES	(2, 'Thomas', 'Dupont')
;
INSERT INTO tUsers	(id, firstname,	lastname)
VALUES	(3, 'John', 'Doe')

CREATE TABLE `tStaging` (
  `id` int(11) NOT NULL,
  `firstName` varchar(30) NOT NULL,
  `lastName` varchar(30) NOT NULL,
  `active` tinyint(4) DEFAULT NULL
  )
;
INSERT INTO tStaging	(id, firstname,	lastname, active)
VALUES	(1, 'Clément', 'Perrod', 1)
;
INSERT INTO tStaging	(id, firstname,	lastname, active)
VALUES	(3, 'John', 'Doe', 0)
;
INSERT INTO tStaging	(id, firstname,	lastname)
VALUES	(4, 'Elise', 'Duffin')
;
INSERT INTO tStaging	(id, firstname,	lastname)
VALUES	(5, 'Anonie', 'Mousse')

Using the clause is as simple as follows:

insert into	
	tUsers
select	st.id,
	st.firstName,
	st.lastName,
	st.active
from	tStaging	st
ON DUPLICATE KEY 
UPDATE	active = st.active

OUTPUT clause of the UPDATE statement

This upsert method is based on the OUTPUT clause of the UPDATE statement, which is available in Microsoft SQL Server 2000 and 2005 and above.

First, an update is performed on @tUsers, using the rows of @tStaging. The OUTPUT clause keeps all updated rows in the @tUpdated table. We are now able to see which rows of @tStaging have been used and which have to be inserted.

declare	@tUpdated	table
(
	id		int
)

update	@tUsers
set	active = st.active
output	inserted.id
into	@tUpdated
from	@tStaging	st,
	@tUsers		us
where	us.id = st.id

The second step is a simple INSERT statement, in which rows that are not in @tUpdated are selected from @tStaging using a LEFT OUTER JOIN (or a NOT EXISTS statement).

insert into	
	@tUsers
select	st.id,
	st.firstName,
	st.lastName,
	st.active
from	@tStaging	st
left join
	@tUpdated	up
on	up.id = st.id
WHERE	up.id is null

Final points

In this article, I have been through the most famous solutions to update or insert rows in a table (except the cursor one). Even if some methods are easier to use, it all depends on the installed DBMS.

MERGE statement	ON DUPLICATE KEY clause for INSERT statement	output clause for UPDATE statement	cursors
IBM DB2 Microsoft SQL Server 2008 Oracle 9i and later versions	Microsoft SQL Server 2000 Microsoft SQL Server 2005	MySQL 5.5 and later version	PostgreSQL

Stay tuned: we will discuss later why there are so many differences between DBMS.

Read more about merge

• MS SQL Server 2008: MERGE
• IBM DB2 v9: MERGE
• Oracle 11g: MERGE
• MS SQL Server 2000 and 2005: output
  clause
• MySQL 5: ON DUPLICATE KEY clause

]]> http://rtfblog.com/2011/09/07/update-or-insert/feed/ 1 cperrod http://rtfblog.com/2011/07/27/fix-gallery3-to-extract-iptc-headlines-and-captions/ http://rtfblog.com/2011/07/27/fix-gallery3-to-extract-iptc-headlines-and-captions/#comments Wed, 27 Jul 2011 19:17:20 +0000 Albin Kauffmann http://rtfblog.wordpress.com/?p=405 ]]> Gallery3, a web based photo album organizer, can extract the IPTC.Caption field of photos you import. This caption is then used to set the description of photos. However, this default behavior might not be what you would expect.

Introduction to Gallery and IPTC

Gallery is an open source web based photo album organizer. Gallery3 intends to replace the famous Gallery2. At the day of this writing, Gallery 3.0.2 is the latest release. Please find more information on the website.

The IPTC IIM is a standard defining a set of metadata directly saved in photo files and thus allowing the portability of photo information and the compatibility between photo software. More information on IPTC can be found in the resources.

Change Gallery’s default behavior

The current stable release (Gallery 3.0.2) only extracts the IPTC.Caption field of photos you import. However, the world is imperfect and photo software are not using the same IPTC fields to designate the title and the description of a photo. As far as I know, metadata fields should match:

IPTC.Headline or IPTC.ObjectName => Title
IPTC.Caption or IPTC.Comment => Description

As I share my Gallery3 space with other users (who are potentially using different software) and that we usually set only a title to pictures, I’d like Gallery to automatically find and set the photos titles. In other words:

if IPTC.Headline or IPTC.ObjectName is set
   photo.title = (IPTC.Headline or IPTC.ObjectName)
   if IPTC.Caption or IPTC.Comment is set
     photo.description = (IPTC.Caption or IPTC.Comment)
else if IPTC.Caption or IPTC.Comment is set
   photo.title = (IPTC.Caption or IPTC.Comment)

You’ll find my patch here. Enjoy it! I did not send this code to the official project because developers don’t seem to agree on IPTC tags use (see the tickets in resources).

Resources

• My patch
• trac tickets concerning this subject:
  • Ticket 608
  • Ticket 499
• IPTC norm:
  • IPTC standard for Photo Metadata
  • Metadata, formats and applications compatibility

]]> http://rtfblog.com/2011/07/27/fix-gallery3-to-extract-iptc-headlines-and-captions/feed/ 0 albinkauffmann http://rtfblog.com/2011/06/27/recursion-in-sql/ http://rtfblog.com/2011/06/27/recursion-in-sql/#comments Mon, 27 Jun 2011 20:56:51 +0000 Clément Perrod http://rtfblog.wordpress.com/?p=369 ]]> We are used to easily writing recursive calls and loops in procedural languages, but this doesn’t hold true in SQL. Indeed, it depends on the DBMS used.

This article will focus on the ability to write recursive queries using Common Table Expressions (CTE). CTE are an addition from the SQL 1999 standard and are not yet available in all DBMS. As of today, CTE are at least compatible with:

• Microsoft SQL Server 2000 / 2005 / 2008
• IBM DB2
• Oracle 9.2
• PostgreSQL

Definition and syntax

A CTE is a temporary view with a very short life. It is only available in the SELECT following the declaration of the CTE, not longer!

The definition and the declaration is done in the same block, introduced by the WITH statement.

WITH expression_name [ ( column_name [,...n] ) ]
AS
(
   CTE_query_definition
)

/* You can use the CTE in the following select */
SELECT <column_list>
FROM   expression_name;

/* And this is it. The following select won't work */
SELECT <column_list>
FROM   expression_name;

As stated in the MSDN:

Using a CTE offers the advantages of improved readability and ease of maintenance of complex queries. The query can be divided into separate, simple, logical building blocks. These simple blocks can then be used to build more complex, interim CTEs until the final result set is generated.

Beware: the only situation in which CTE improve the speed of an SQL request is when they are used more than once per request. In this case, the CTE are evaluated only once. Otherwise they only improve its readability and maintainability. And even if it is a key point for any developper, the CTE feature I want to introduce here is the recursivity.

WITH expression_name [ ( column_name [,...n] ) ]
AS
(
   CTE_query_initialization

   UNION ALL

   CTE_query_recursion
)

A recursive function needs a base case, an inductive step and a stopping criterion. The initialization is done in the first SELECT while the inductive step is defined in the second one. Finally, the stopping criterion will be part of the WHERE statement of the second condition.

With such a feature it is now possible to dynamically build datasets and avoid table creation for example (as shown in the next part of this article).

Basic example

Before going further, a simple example may help to understand the power of CTE in SQL. The following code retrieves the value of the factorial function

Reminder:

• factorial (0) = 1
• factorial (1) = 1
• factorial (n) = n! = n * factorial (n -1)
• factorial (4) = 4! = 4 * factorial (3) = 4 * 3 * 2 * 1

The following code returns the value of factorial (4). Amazing, isn’t it?

/* Code tested with MS SQL Server 2005/2008 */
with		cteFacto
as
(
   select	1			as I_FACTO,
			1			as FACTO
   union all
   select	I_FACTO + 1		as I_FACTO,
			(I_FACTO + 1) * FACTO	as FACTO
   from		cteFacto
   where	I_FACTO < 9
)
/*
    I_FACTO indicates the factoriel we want
    FACTO retrieves its value
*/
select		FACTO
from		cteFacto
where		I_FACTO = 4  /* = 24 */

Generate datasets with dates

Dates manipulation is another, closer to concrete use cases, but still simple example of CTE in SQL.

Using CTE and recursivity, it is now very easy to generate this kind of dataset:

AGENDA	                year    month   day
2011-01-01 00:00:00.000	2011	1	1
2011-01-02 00:00:00.000	2011	1	2
2011-01-03 00:00:00.000	2011	1	3
...
2011-01-27 00:00:00.000	2011	1	27
2011-01-28 00:00:00.000	2011	1	28
2011-01-29 00:00:00.000	2011	1	29
2011-01-30 00:00:00.000	2011	1	30
2011-01-31 00:00:00.000	2011	1	31

And, it is as simple as this:

/* Code tested with MS SQL Server 2005/2008 */
declare		@startingDate	as	datetime
set			@startingDate	=	convert (datetime, '2011-01-01', 121)

/* Semicolons are mandatory when the with statement is not immediately after the go keyword  */

;with		cteAgenda
as
(
	select	@startingDate				as AGENDA
	union all
	select	DATEADD (day, 1, AGENDA)	as AGENDA
	from	cteAgenda
	where	agenda < convert (datetime, '2011-02-01', 121)
)

select		AGENDA,
			YEAR (AGENDA)	as year,
			MONTH (AGENDA)	as month,
			DAY (AGENDA)	as day
from		cteAgenda
where		AGENDA < convert (datetime, '2011-02-01', 121)

This kind of code may avoid the creation of a one-shot table by pushing values in a temporary table. Which, in my opinion, is a real benefit.

Final points

CTE are more than simple temporary views: they also allow the use of recursivity in SQL. Using CTE with temporary tables helps to create datasets that can be used in testing environment. They can also be used to explore datasets, or to recursivly apply operations on tree-structured datasets.

Read more about CTE

• MSDN: CTE (EN)
• MSDN: Recursive CTE (EN)
• PostgreSQL: CTE (EN)
• Oracle 9i 2: CTE (EN)
• IBM DB2: CTE (FR)
• CTE presentation (EN)
• CTE Recursives (FR)

]]> http://rtfblog.com/2011/06/27/recursion-in-sql/feed/ 0 cperrod http://rtfblog.com/2011/05/25/taking-advantage-of-multithreaded-environments-with-ruby/ http://rtfblog.com/2011/05/25/taking-advantage-of-multithreaded-environments-with-ruby/#comments Wed, 25 May 2011 19:11:18 +0000 Xavier Noëlle http://rtfblog.wordpress.com/?p=315 ]]> If you have ever tried to write parallel scripts / applications using Ruby’s threads, you may have stumbled over the fact that, sometimes, some cores are left unused. You may be especially sad if you notice that your brand new war-machine, with 2 physical cores or more, is not fully exploited.

Then, you would be right to wonder if using Ruby and taking advantage of multicore (i.e. writing multithreaded applications) are compatible.

The simple answer is yes. Let’s dig into the more complicated answers: we’ll first explain how Ruby’s threads work, then explain why they sometimes seem to be broken, and eventually suggest four efficient solutions to get rid of their limitations.

Ruby threading models

As explained in my previous article about Ruby’s speed, the 1.8.x and 1.9.x branches of Ruby are coexisting since 2008. One of the most important performance tweaks between the two mainstream interpreters¹ is related to the threading model itself.

Prior to the 1.9.x branch, Ruby, with MRI, was using green (user-level) threads. With this threading model, one can map only one application per kernel-level scheduled entity (simply put, an application can use only one kernel thread); thus, the kernel does not even know if your application is multithreaded or not. This approach has several advantages, such as:

• providing threading on environments which do not support threading (it is handled by the application itself);
• reducing the cost of context-switching (switching between threads).

The choice of using green threads was made for Ruby 14 years ago, for compatibility reasons, when a lot of systems did not provide threading support.

But the major drawback of this threading model is that it cannot benefit from multi-core processors. This also means that threads may block each other during I/O operations, because they share the same execution context.

With the release of YARV, Ruby 1.9.x now uses kernel threads; this means that every single Ruby thread is mapped to a kernel thread and can be run on a different core, and thus enables a single Ruby script to use more than one core.

The Global Interpreter Lock (GIL)

The latter model sounds perfect and allows scripts to run more efficiently in a lot of concurrency situations.

But, everything is not perfect in the shiny kernel threads’ world: the Python and Ruby lovers among you may have already heard about the Global Interpreter Lock, a huge mutual exclusion lock (aka mutex), preventing multiple Ruby threads to use the interpreter at the same time.

Two Ruby threads may then not run pure Ruby code (more precisely, use its API) at the same time. This problem is illustrated by running the following code:

2.times() do
  Thread.new() do
    a = Array.new()
    a << 1
    a.clear()
  end
end

This code starts two threads which will perform an array instantiation, append the “1” value to it, and then empty it. Let’s schedule this by hand, in a pure didactic and fictitious way; red (resp. green) blocks represent periods of inactivity (resp. activity):



What is striking in this figure is the forced inactivity of the B thread when the A thread is running and vice-versa.

The problem is different when some pure C code is involved. As stated in the previous article, Ruby features easy interactions with C. Then we may simply add a call to some random C code (provided that it explicitly releases the GIL, and then doesn’t use the Ruby API), right after the array instantiation, and the figure becomes:



Much more interesting! This illustrates clearly that the problem only occurs when the Ruby API is involved.

Getting rid of the Global Interpreter Lock (GIL)

There are four main solutions to the GIL restriction (two of them are the subject of in-depth articles, which will be linked directly below):

• Run your script on a different VM: for example, JRuby and Rubinius have gotten rid of the GIL, so you might want to try them out.
• Wait for a new mainstream VM: Koichi Sasada (the creator of YARV) announced that he may be working on a multi-VM version of the interpreter, in which the GIL would no longer exist… we’ll stay tuned about that!
• Write long and blocking parts in C: extensions can run in parallel, given this simple constraint: you must manually ask the Ruby VM to release the GIL for the current thread. But after doing so, you shall not call, directly or not, any function of Ruby’s API, since this would lead to an undefined behavior. Some gems take advantage of this solution, such as mysql2, which allows to execute Ruby code while waiting for the result of a MySQL request (the mysql gem does not allow that).
• Use separate processes: if you want to get rid of the GIL, run multiple interpreters (quite straightforward)! Ruby ships with a good IPC library named DRb, which makes tasks distribution really easy.

Closing comments

This article tried to depict what you can expect from Ruby in terms of parallelism. It was mainly aimed at convincing you that green threads in Ruby 1.8, or the GIL in Ruby, are no show-stoppers, even if they require a little adaptation. If you’re curious about the last two solutions mentioned above, you are strongly encouraged to stay tuned for the in-depth articles:

---

1. We won’t cover other interpreters’ implementation in this article.

]]> http://rtfblog.com/2011/05/25/taking-advantage-of-multithreaded-environments-with-ruby/feed/ 0 xaviernoelle RTFBlogRubyGIL RTFBlogRubyGILmysql http://rtfblog.com/2011/04/11/introduction-sba/ http://rtfblog.com/2011/04/11/introduction-sba/#comments Mon, 11 Apr 2011 08:49:17 +0000 Clément Perrod http://rtfblog.wordpress.com/?p=68 ]]> Search Based Applications (SBA) are applications based on search engines. They are designed for users who want to discover and analyse information. They usually require processing large amounts of data, processing data in real-time or handling unstructured or semi-structured data, among others. To handle these requirements, they benefit from the appropriate processes already used by search engines (scalable foundations and natural language processing, for example).
Input data can be structured (databases…) but also unstructured (Office documents, PDF, mails…) or even coming from other applications (ERP, CRM, intranet, extranet…).

Unlike traditional search engines, SBA provide ad-hoc interfaces to data (dashboards, mashups…), so that the user can access the right information without necessarily having to perform requests manually (in the context of BI, for example, it allows him to take the good decision on time by giving him a relevant view of the data).

Notice the difference between discovering and searching: when searching, the user knows what he is looking for; when discovering, he starts by searching something, which brings him to a final point he may not have expected. Think about it: discovering is what we need when taking decisions. Why? Because even if we know what information we need we may not know how to get it. Discovering helps users to take better decisions.

The players

The main SBA platform editors are also Information Access market players. Most of them have more than five years of experience in providing indexing tools to their clients, which gives them some credibility.

The current leader is Endeca with Endeca Lattitude, which is a big American company also operating in Europe and Asia. So they have clients and offices all over the world. Then comes Attivio with the AIE Platform. The main difference is that Attivio does not provide only a web client but also a desktop one where any user can write SQL-like requests on any kind of data. Two french companies are also in the SBA market. The biggest one is Exalead with Exalead CloudView 360. They are mainly doing business in the French market but are trying to grow up outside the Europe (even more since Dassault System bought them). Finally, the last one and the smallest: Sinequa. It is a young company 100 % focused on the SBA market.

The following “Magic Quadrant” comes up as a summary of the different editors.



General Anatomy of an SBA

Three steps are present in a SBA project, whatever the chosen solution is (at least it is true for Attivio, Endeca and Exalead).

First, the appplication needs to collect data by connecting to all sources. Fortunately, SBA provides APIs and tools to make this easy. As of today, connectors have already been developed for:

• Databases (through JDBC/ODBC)
• Filesystems
• Internet / extranet / intranet
• E-mails
• LDAP
• ERP, CRM, CMS

If a connector does not exist, it is possible to create it or ask for it. In this part of the project, it is also possible to transform unstructured data in semi-structured one by using semantic and analytic processes.

Then comes the indexing tool. This part is the easiest one because it does not require any development on the client site. This is where the search engine experience and intelligence comes in: the application automatically knows how to collect all data according to the configuration and indexes it, in order to be able to retrieve all information when needed.

The project ends with the restitution part. Building dashboards and pages is almost as important as connecting to data. Once again, editors provide APIs and tools for this part, so small effort is needed to produce interesting and efficient layouts.

Generally, when a client starts an SBA project, he only has an idea of the expected result and the needs evolve with it. Dealing with such constraints is possible because these projects are based on Agile software developments. According to Exalead experience in this kind of projects, the average needed time is around a year with loops of 2 months.

Going further

The SBA market is still very young but it may become a huge one in the future. For example, clients with BI products are asking for such projects to complete their BI solution.

If you are looking for complementary information about SBA, you may also search for UIA (Unified Information Access) or UBI (Unified Business Information).

The following articles may also give you more information:

• The BI-Search evolution (Endeca) (EN)
• BI Search : Implementation Considerations (Information Builders) (EN)
• Search-based applications support critical decision-making (EN)
• Who Needs Unified Information Access, Anyway? (Attivio) (EN)
• Méthodes AGILES : une belle définition (FR)
• Wikipedia article (EN)

]]> http://rtfblog.com/2011/04/11/introduction-sba/feed/ 1 cperrod SBA Magic Quadrant http://rtfblog.com/2011/03/21/why-you-should-probably-not-bother-about-ruby-s-speed/ http://rtfblog.com/2011/03/21/why-you-should-probably-not-bother-about-ruby-s-speed/#comments Mon, 21 Mar 2011 22:42:43 +0000 Xavier Noëlle http://rtfblog.wordpress.com/?p=9 ]]> The expressivity and power of Ruby make it a really efficient programming language. Its dynamic nature results in lots of enjoyable benefits (introspection, dynamic class reopening…); it is also responsible for a part of its performances, which are sometimes believed to be poor.

Is Ruby really slow? While being obviously slower than most natively compiled languages, is the performance gap really worth noticing and should it prevent you from using this great language?

If all you’re coming for is pure performance benchmarks, you may directly skip to the last part of this article (“Lies, damned lies, and benchmarks”), since this article does not aim at benchmarking Ruby.

ruby –version

Depending on your OS or the way you installed Ruby on your computer, you may be facing two different major releases of Ruby. If you take a look at Ruby’s history, you can notice that the 1.8.x branch started in 2003 and was joined by the 1.9.x branch at the very beggining of 2009. Since then, these two versions are coexisting, while being different in a number of ways.

The most interesting change is related to the interpreter itself: YARV, the mainstream Ruby 1.9.x interpreter, is now dealing with bytecode, rather than constantly parsing and executing unoptimized Ruby code. The addition of a few other optimizations induced a great speed boost for most Ruby scripts (about three times faster on average, according to benchmarks which are legion on the Web: for example, here and here).

Sadly, most beliefs about Ruby are based on the old 1.8.x branch, which was the most used branch until now, but tends to be replaced by newer versions; a great effort is being made to migrate major gems (Ruby packages, such as Rails) and applications to the 1.9.x branch. The new Ruby interpreter comes with a revision of the language itself, which makes this migration not as easy as it seems and explains why some people are still late to the party.

One language to rule them all, and in the runtime bind them

Aside from the well known MRI and YARV, which are respectively shipped with Ruby 1.8.x and Ruby 1.9.x, some other mature Ruby interpreters are available and quite widely used. They feature some additional or structural refinements, which make them better suited than the mainstream interpreters in some situations. Two of the most notable challengers are:

• JRuby: an interpreter based on the Java Virtual Machine, providing an easy integration with Java libraries as well as the obvious advantages of running inside the JVM;
• Rubinius: an interpreter written in C++, which features Just-in-time (JIT) native machine code compilation.

These interpreters outperform the standard interpreters for some specific problematics (one of them being parallelism efficiency, which will be the main topic of an upcoming article) and may be worth benchmarking when performances become a concern, before deciding to drop Ruby.

C for hostpots, Ruby for everything else

Most of the standard Ruby library, as well as a number of gems (Ruby packages), is written in pure C. Indeed, Ruby features easy interactions with C and C++: you can use the Ruby API in your C code and your Ruby code can, in turn, easily retrieve results from your C code. This is roughly achieved by using one or more C files, a file composed of a few lines of Makefile-like syntax which will build a library for you, a function call and… that’s it!^[1]

Knowing that, it would be wise to reconsider Pareto’s principle: most of the time, 20 % of your code is responsible for 80 % of your application’s runtime. Rewriting these slow parts in a compiled language like C, while keeping structural parts human-readable by using a high-level language should be the way to go, when possible (i.e. most of the time); that is exactly the Ruby way of thinking. Moreover, when profiling your applications, you will often notice that the slow parts of your application are completely independent from the language you’re using.

And believe me: most of the time, you won’t even need C at all! I have myself never really needed to rewrite parts of my applications in C, even when raw performances were a vital concern. In fact, the only time I really thought I needed it led me to a complete waste of time, since I was trying to improve performances by focusing on the wrong problem. This is not to say “never use C” but rather “use C when all other reasonable options failed”; mixing code may reduce your source code readability and maintainability.

Lies, damned lies, and benchmarks

If you expected benchmarks in this article, you will be disappointed. The only general truth that should come to your mind is the following ranking^[2] (sorted by decreasing performance):

• Natively compiled languages (C, C++, Fortran, Pascal…)
• Byte-compiled languages (Ruby 1.9.x, Python, Java…)
• Interpreted languages (PHP, Shell-scripting languages, Ruby 1.8.x…)

Everything else is usually a waste of time for general applications, especially when you are allowed to mix compiled chunks of code in your Ruby or Python application. Keep in mind that the time you save by using a more powerful (in terms of expressivity, readability, standard structures…) language can be used to throw up some heavy algorithmic as well as parallelization optimizations, which will make a greater difference, most of the time.

Well, if you really want benchmarks (because I know you’ll be upset otherwise), you may want to follow this link (but I warned you!).

Conclusion

I hope this article convinced you that considering Ruby as a slow language is mostly false and generally irrelevant. For those of you who need extreme performances, a natively compiled language like C or C++ will certainly perform better in some cases.

But if you don’t fit in this category and, nevertheless, plan to drop Ruby^[3] just because you think it won’t be fast enough for you, you may want to reconsider your thoughts…

---

1. This will be discussed more thoroughly in an upcoming article about Ruby and parallelism.
2. Languages are categorized with respect to their current main implementation (eg. MRI for Ruby 1.8, YARV for Ruby 1.9…).
3. The same reasoning applies to Python as well.

]]> http://rtfblog.com/2011/03/21/why-you-should-probably-not-bother-about-ruby-s-speed/feed/ 4 xaviernoelle http://rtfblog.com/2011/03/12/improve-your-internet-upload-bandwidth-management-by-configuring-your-linux-router/ http://rtfblog.com/2011/03/12/improve-your-internet-upload-bandwidth-management-by-configuring-your-linux-router/#comments Sat, 12 Mar 2011 17:48:41 +0000 Albin Kauffmann http://rtfblog.wordpress.com/?p=87 ]]> If you are running a Torrent, FTP or whatever kind of server behind your DSL link, you have probably noticed that interactive sessions get really slow (Web browsing, SSH…). As a first solution, you can of course configure your Torrent or FTP server to limit its upload but it’s not fully optimized as this wouldn’t use 100 % of your upload bandwidth.

(French version of this article)

Before introducing a solution, let’s briefly explain what is happening. Typical Ethernet modems handle packets with the same priority. Only IP packets with a specific “Type of Service” (ToS) benefit from a special treatment and are sent in priority. This allows services like VoIP to stream data in real-time.

The goal of this article is to show how a Linux router can be configured to control your upload bandwidth by setting up different priorities to the network traffic. This task is achieved by using the Linux traffic control functionality (tc).

Controlling your download traffic, while possible with TCP connections, is much more complicated. Indeed, you can’t access your ISP routers in order to set up your own configuration. Then, your only possibility is to drop received packets in order to decrease the speed of TCP connections. This works because the TCP protocol automatically adapts the speed of a session. This article doesn’t get into download bandwidth control but you should find interesting links in resources. Finally, remember that ISP are implementing QoS so you don’t have to worry about the priority of your VoIP/RTP/… connections.

Overview

In this article, I suppose a Linux box is already configured as a router:



Note: the IPv6 configuration is not covered in this article but it should be pretty similar.

In order to control the order in which packets are sent to the Internet, packets need to be queued and sorted at the Linux Box level:



By default, packets are queued at the modem level, that’s why this article firstly shows how to force the queuing to be done by the Linux box. Then, tc is mentioned to sort packets and iptables to mark them.

Force queuing at the Linux Box level

The first thing to do is to configure the Linux Box so that it fully controls the outgoing bandwidth. As already mentioned, outgoing traffic needs to be queued by the Linux Box and not by the DSL modem.

Unfortunately, the only way is to limit the Linux Box upload bandwidth so that the modem queue never gets filled (in fact, there might exist some modem allowing to set the size of the queue but I’m not even sure it will help).

To do so, Linux offers a Traffic Control service (tc). tc allows to define a tree of qdiscs (where packets are actually queued) and classes. A class contains one or several qdisc(s) and allows to define where/how packets are queued. In this article, two qdiscs are going to be used:

• HTB (Hierarchy Token Bucket): HTB allows to shape network traffic;
• SFQ (Stochastic Fairness Queueing): SFQ allows packets of each session to be send in turn (all sessions are then treated equally).

For more information, please read the man pages of tc, tc-htb and tc-sfq.

So, here are the commands to limit your upload to $RATEUP ($DEV == eth1):

ip link set dev $DEV qlen $QLEN mtu $MTU

tc qdisc add dev $DEV root handle 1: htb default 1
  tc class add dev $DEV parent 1: classid 1:1 htb rate ${RATEUP}kbit
    tc qdisc add dev $DEV parent 1:1 handle 10: sfq perturb 10

In a few words, these commands:

1. decrease the size of the queue and set the MTU to something more appropriate for the DSL connection ($QLEN=100 and $MTU=modem_value);
2. replace the default queue by a HTB qdisc;
3. add a class to the HTB qdisc to limit the bandwidth to ${RATEUP}kbit;
4. add a SFQ qdisc to this class.

Sort packets with tc

The HTB class system allows to define several queues (qdisc) with different priorities. This way, packets waiting to be sent are sorted and sent (the higher priority is 0):

tc qdisc add dev $DEV root handle 1: htb default 12
  tc class add dev $DEV parent 1: classid 1:1 htb rate ${RATEUP}kbit

    tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${RATEMIN}kbit \
      ceil ${RATEUP}kbit prio 0
    tc class add dev $DEV parent 1:1 classid 1:11 htb rate ${RATEMIN}kbit \
      ceil ${RATEUP}kbit prio 1
    tc class add dev $DEV parent 1:1 classid 1:12 htb rate ${RATEMIN}kbit \
      ceil ${RATEUP}kbit prio 2
    tc class add dev $DEV parent 1:1 classid 1:13 htb rate ${RATEMIN}kbit \
      ceil ${RATEUP}kbit prio 3

      tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
      tc qdisc add dev $DEV parent 1:11 handle 11: sfq perturb 10
      tc qdisc add dev $DEV parent 1:12 handle 12: sfq perturb 10 # def. queue
      tc qdisc add dev $DEV parent 1:13 handle 13: sfq perturb 10

tc filter add dev $DEV parent 1: prio 0 protocol ip handle 10 fw flowid 1:10
tc filter add dev $DEV parent 1: prio 0 protocol ip handle 11 fw flowid 1:11
tc filter add dev $DEV parent 1: prio 0 protocol ip handle 12 fw flowid 1:12
tc filter add dev $DEV parent 1: prio 0 protocol ip handle 13 fw flowid 1:13

This script creates 4 queues with priorities going from 0 to 3 and with a minimum rate ($RATEMIN). This minimum rate allows low priority applications to always have access to the Internet, that is applications won’t fail with a network timeout because another high priority application is using the whole bandwidth.

The queue defined with the classid 1:12 is the default one and is used if no tc filter command defines another behavior. In this example, tc filter commands allow to sort packets according to the tag previously set by iptables (then, untagged packets use the default qdisc). The next section explains how packets are tagged.

Mark packets with iptables

If you already know how to use iptables, marking packets is really easy. Rules must be added to the mangle table and to the POSTROUTING chain. Here is a small example which shows how packets could be sorted with the previous tc configuration:

# High priority for ping packets
iptables -t mangle -A POSTROUTING -p icmp -j MARK --set-mark 10

# Default for low port outgoing connections (clients)
iptables -t mangle -A POSTROUTING -p tcp --dport 0:1024 -j MARK --set-mark 11

# Default for low port ingoing connections (servers)
iptables -t mangle -A POSTROUTING -p tcp --sport 0:1024 -j MARK --set-mark 12

# High priority for DNS, SIP and ACK packets
iptables -t mangle -A POSTROUTING -p udp -j MARK --set-mark 10
iptables -t mangle -A POSTROUTING -p tcp -m length --length :64 -j MARK \
  --set-mark 10 # Small packets are generally ACKs
iptables -t mangle -A POSTROUTING -p tcp --dport sip -j MARK --set-mark 10
iptables -t mangle -A POSTROUTING -p tcp --dport sip-tls -j MARK --set-mark 10

# High priority for interactive sessions
iptables -t mangle -A POSTROUTING -p tcp --dport ssh -j MARK --set-mark 11
iptables -t mangle -A POSTROUTING -p tcp --sport ssh -j MARK --set-mark 11

# Low priority for FTP data and torrent outbound traffic
iptables -t mangle -A POSTROUTING -p tcp --sport MINPORT:MAXPORT -j MARK \
  --set-mark 14
iptables -t mangle -A POSTROUTING -p tcp --sport 6881:6999 -j MARK \
  --set-mark 14

The configuration is now finished but you need to run tests in order to determine the best value of $RATEUP (tc script). To do so, run a ping to a remote host and a program which uploads data. As long as the ping stays low, you can try to increase $RATEUP. If the ping is high, you need to decrease $RATEUP.

Conclusion

You can now impress your friends by keeping a small ping while running a lot of services behind your DSL connection

tc offers a lot more possibilities that can be found by reading man pages and links given in the Resources section. You can also download my tc-restore script which sets the tc configuration on a Debian like system.

Resources

• My tc-restore script
• man pages of tc, tc-htb, tc-sfq…
• http://lartc.org and especially http://lartc.org/howto/lartc.qdisc.html
• http://tldp.org/HOWTO/Traffic-Control-HOWTO/index.html

]]> http://rtfblog.com/2011/03/12/improve-your-internet-upload-bandwidth-management-by-configuring-your-linux-router/feed/ 2 albinkauffmann tc-overview tc-overview-queue http://rtfblog.com/2010/12/15/hello-world/ http://rtfblog.com/2010/12/15/hello-world/#comments Wed, 15 Dec 2010 16:13:42 +0000 Xavier Noëlle http://rtfblog.wordpress.com/?p=1 ]]> The RFTBlog is coming soon…

Stay tuned and follow us also on our Facebook or Twitter page for the blog launch!


]]> http://rtfblog.com/2010/12/15/hello-world/feed/ 0 xaviernoelle