Understanding Data Analysis: Part Two

Data analysis is a basic skill for any business personel at any modern enterprises. Most of the decision making processes at such organizations lay on a solid analytical background before they reach into the final resolution. Almost all the modern Business Intelligence technologies originate from requirements at the data analysis steps. Learning the data analysis process and concept gives very good meaning for understanding why and how the BI technologies have evolved into the current state.

I have a series of blog entries on data analysis concepts, process, and techniques. This is the second part of the series. This blog entry is motivated and actually based on the following references.

1. Michael Milton: Head First Data Analysis

 

Part II: Test, Optimize and Visualize

While following the four steps described in my previous entry, every analyst is facing the problem of defensing her/his decision-input by the hard, bloody reality of the real world result. All analytical models, decisions and recommendations have to survive through the examinations when they are deployed and used in the real world. Time proves the truth.

But what about fine-tuning and adaptive adjustment to a defined analytical (or mental) model? The world evolves (if not revolutionalizes), so does the mental models. How to give the analysts opportunities to adjust the models and improve them?

Test, test, and test. Yes, test your models. And by "test" I am not saying using just simulations and paper-based data and examples (made by empirical data). You must test your model in the real world and do the experiments in a real scenario.

In many real world scenario, doing test gives very good sense to solve the problem in a sustainable manner. Making a well-planned and well-executed experiments gives a powerful proof of the analyst's judgements. How to carry out the experiment? The very basic element of experimenting is to do surveys. Making surveys before and after a planned test activity can let the analyst have more insight on how effective the test activity has been.

One good example in Michael's book is to use a few local coffee shops to experiment the analytical models which can contribute to a national wide deployment to all the coffee shops. Before running the experiments, the analyst must define what are the factors to observe and compare. And a baseline data must be prepared in order to compare the results. A set of control groups, in contrast to experimental groups, must be established to ensure the comparison. Selecting the control groups requires the analysts to be very careful. The analyst must make sure that other confounders are not blending into the experiment and make sure that all the control groups are "equally" the same on the confounders. After the groups are defined, the rest of the steps are simple and direct. One just need to execute the experiments and collect result for comparison. And the final conclusion will be clear.

What is confounder, precisely? A confounder is a difference among the target group of the analyst's research. A confounder is a factor that, if you use it to compare, makes the analysis result sensible compared to other factors which are non-confounders. A confounder can be factors such as locations, age groups, gender type, etc.

Doing data analysis with identified confounders is not very difficult. The analyst just needs to use the confounders to break out data into smaller chunks to test the result. Identifying confounders based on a given set of data is not easy. Analysts can typically use a set of mathematical or statistical tools to work it out.

A very typical method, when the analytical model or the scenario for analysis gets complicated, is to define a mathematical or statistical model and to optimize it. One thing to make it clear is that no mathematical model can be made so perfect that all real world factors can be modelled, represented and "mathematicalized". One always has to drop those that are too complicated and focus on the "main thing."

The way that an analyst tries to solve the optimization is to define a mathematical model for the problem scenario and then solves the problem by classical mathematical methods. When it happens that the results do not show to be the most optimal result, the analyst must find out where in the model was set to a wrong assumption or if anything should have been included in the model or if it is the right mathematical model to use. There is no perfect model for all scenarios. The analyst's role is to choose the right one for the specific scenario.

Optimization can be done when it is possible to represent the main factors using a mathematical model. There are many data analysis tools (software packages) that can help a data analyst to find out the most optimal result. The Solver package in Excel is a good example of it. And I shall also list R, SAS, SPSS, Mathematica, Matlab and so on.

To solve an optimization problem, the analyst must define an objective function, find out all constants, variables, and search for other constraints related to the identified variables. In many cases, it brings extra benefit if a visualization of the objective function, together with the constraints, can be drawn on a two or three dimensional space. Visualization gives the best chances to verify the analysis and estimate the most optimal result.

In fact, visualization has been broadly used in the scientific (or, academical) world to understand observations and facilitate key decisions on research directions. The business world (and the BI industry) is following this trend in recent years. Good examples are the population of visual analytics tools such as SAS Visual Analytics, Tableau, and Qlikview.

Data visualization means to show your data with diagrams instead of tables of numbers. Visualization is important and extremely useful in most scenarios. It is equally to say that finding out the right visualization is also vital and decides how the clients receive and accept the observations and conclusions.

Using visualizations does not mean that the analyst needs to make fancy or beautiful diagrams. In fact, the visualization should be clear,simple and straightforward.

How to find the best visualization to pinpoint the analysis? A recommended way is to use scattetplots. Just start by choosing an X and Y dimensions and start drawing the data points. Very often the analysts can find many interesting patterns and are able to define new dimensions. A standard analysis would use independent variables as the X axis and dependent variables as the Y axis. Of course there will be more than one dependent and independent variables. The analysis report normally involves diagrams with different combinations of variables which can help to identify certain patterns in the data points.

There exists many excellent data visualization tools. Besides Excel, Tableau and other commercial tools, open source tools, such as R, can be a very good choice.

 

在Blogsy发表的

Understanding Data Analysis: Part One

Data analysis is a basic skill for any business personel at any modern enterprises. Most of the decision making processes at such organizations lay on a solid analytical background before they reach into the final resolution. Almost all the modern Business Intelligence technologies originate from requirements at the data analysis steps. Learning the data analysis process and concept gives very good meaning for understanding why and how the BI technologies have evolved into the current state.

I have a series of blog entries on data analysis concepts, process, and techniques. This is the first part of the series. This blog entry is motivated and actually based on the following references.

1. Michael Milton: Head First Data Analysis

 

Part I: The basic steps

Making decisions at a management position can be either easy or difficult. Using intuitions and always bet on one's luck has been a tradition for many "cowboy" style managers. To keep being successful and stay at the right career steps, many managers choose to use decent approaches and make decisions based on a set of rational and serious analysis. This starts the era of data analysis.

No matter what analysis it is, the very basic steps of any analysis are to define the problem, break it down, take observations and make partial solutions, assemble the results and make the final conclusions. Data analysis is about applying the analysis with a large set of data at hand.

The information age brings massive amount of data to every business entity. Data means value and opportunities if you know how to understand and use it. In many situations we found ourselves working as a data analyst without knowing the basic techniques and practices about this role. A data analyst knows how to break down a large set of data and give it a structural understanding so that the data becomes intelligence and insights. With a powerful toolbox, a data analyst transforms data into knowledge which often pushes forward decisions to be made. Thus, the basic skill or instinct of a data analyst is the capability to understand and structure data.

Given a scenario or just a piece of data, the typical steps that every data analyst will go through can be summarized below.

1. Define the problem

This is the very first step. Know your problem is the basic instinct for a data analyst. All analysis is targeted at a goal, that is, the question that should be answered or the problem that should be solved. Without defining the question or the problem, the analysis will head for nothing. How to define the problem? Just ask! You should of course ask the person who will make decisions based on your analysis. Sometimes it can just be yourself if you are to make the decision. Don't just ask for the final goal of the problem. Ask what the decision maker means.

In many scenarios, the decision maker needs the help from a data analyst because she/he

- Do not know about the data

- Know somethinng about the data but is not totally sure with it

- is not sure about the problem

- is not sure about the scenario of the problem

- Is not good at making decisions

- Use more intuition than analytics

The data analyst should start by clearly defining the problem with her/his clients. It can be that the client brings more problems or requirements after this step, which is totally fine. Changes to requirement may bring little overhead in terms of administration effort. But it also helps to narrow down the problem in many cases. A lot of the times, to help the clients to identify her/his problem is the very first and the most important task for the analyst. Many clients are even not aware of the problems that they are facing.

There is, of course, a method or concept called "exploratory data analysis." That means the analyst must find valuable hypotheses from the data which worth evaluating. This is already more than a concrete problem.

2. Disassemble the scenario

To find a solution to a problem scenario, a very typical approach is to divide the problem and scenario into small pieces and analyze. This step requires the data analyst break down the problem into smaller pieces in order to start the detailed analysis. It is important to keep breaking down to the level that best fit the analysis.

To break down the problem, one approach is to create a kind of tree structure, starting with the problem defined in the previous step and extending to different sub-nodes until you reach the leaf level.

This also applies to the data. The data analyst will need more detailed data if the first step only provides a general level summary of the data. More data brings more room for analysis and testing of conclusions.

What is the most important technique to use when looking at data? Making comparisons! In fact, making good comparisons is at the core of data analysis. A data will always find it useful to break down the data by finding good and interesting comparisons of data.

As a product from this step, I can imagine that the data analyst completes with a document of two sections. The first is the break-down structure of the problem. The second is a set of rules on how to divide and slice the data.

As it always says, you cannot move the whole mountain all at once, but you can move it piece by piece.

3. Evaluate your conclusion

This step evaluate all the observations from the previous steps. The key to evaluate is to compare. With all the problems on the left hand and all the observations on the right side, compare, compare and compare!

A compulsory to this step is to insert the analy her/him-self to this process. This means the analyst bring her/his responsibility on board and focus on getting the best value for the client as well as the analyst her/him-self. While the analyst is indeed betting on her/his credibility, there comes also more likelihood that the analyst will get more trust and belief from the clients.

I'd like to quote this paragraph from the book Head First Data Analysis.

"Whether you’re building complex models or making simple decisions, data analysis is all about you: your beliefs, your judgement, your credibility"

4. Make the decision

At this step, the analyst summarizes the conclusion and gives the final decision or recommendation. All the analysis results must be formed into a format that is easy for making decisions. Otherwise, the analysis was useless.

The analyst must make it simple and straight for the decision-makers. It is important to get the voice heard (that is, to make it understandable) so that people can make sensible decisions based on the input from the analyst.

One more thing to add is about the analysis report. The analysis report can take as simple as three sections, " background," "understanding of data," "recommendation." And again, it must be simple, concise and direct.

 

With all the steps of data analysis, will we be ready for all kinds of challenges? Not exactly. There are quite a list of things to set ready for. One first question will be "What to do when my analysis turns wrong or incomplete?"

Well, not everything can be so easy and safe. In many cases, the initial data analysis report can get bad feed back and results. What could go wrong then?

The first thing to consider is the assumptions made in the previous analysis. Assumptions are usually based on mental models we built on the scenario. Mental model is how we normally see the world around us. It helps us to understand and interpret information around us. It is like a model in mathematics, where one can build a ton of basic elements and describe the world with them. Very often, the analyst's mental model is affected by the mental model from the clients. So an important lesson for the analyst is to make all the mental models, at least the important parts, explicit to the analysis.

Every part of the analysis, such as the statistic model, the data model, etc. is always dependent on the analyst's mental model. When mental model is wrong, all steps of analysis must be run again to provide a new round of analysis result.

To build the right mental model, the analyst must include both things that are taken as assumption and things that are unknown or uncertain. The "anti-assumption" is very often more important and can lead to new discovery and insights. For example, consider making a list of "things I do not know," "things that I do not know how to do," "things that I have not done," etc. This must be applied to the analysis process and the analyst needs to challenge the client on many aspects of the clients' mental model at the "disassemble" step of the analysis.

And of course, the analyst must ask for new and more detailed data for the new round of analysis. Results from the previous analysis can shed the light on where and how to ask for the new data. A very typical mistake from many analysts is to focus too much on the numerical values (also called "measures") in the data. In fact, the row or column header information is even more important to look at. In many cases, how you drill into and mesh around data decides how the data look like in the end.

Analysis never stops. If the analyst always finds out new clues or observations, she/he can continue the four steps for ever.

 

在Blogsy发表的

A Simple Tutorial on PowerPivot (4)

5. Publishing and Sharing Reports

After doing analysis on the data, it is the time to share the reports and data to other business users. As mentioned early in this tutorial, it is very important to use SharePoint to create a BI collaboration environment in order to protect the Self-service BI from being a pure spreadsheet mart.  

Before publishing PowerPivot workbook to SharePoint, it is normal to consider "beautify" the workbook so that other users find it useful to read and use the report.  For example, it is useful to remove or hide worksheets that are not part of the report, to improve the placement and appearance of the worksheets, or even turn of the grid lines to make it better for using. Another important thing to do is about the configuration of data refresh. The connections that you have at your local PowerPivot for Excel require certain data connection providers, and these connections, after being uploaded to SharePoint, still require the same data connection providers or at least require a re-configuration of the data connections. So it is important to install (or ensure that they are installed) the relevant data connection providers at the server.

When publishing PowerPivot workbook to SharePoint. There is an option to turn on document versioning in SharePont 2010 site. However,  you should be really careful when deciding this option. After turning on the versioning settings, every change to the PowerPivot workbook is saved to the content database. That means many versions (at almost the same size) of the PowerPivot workbook is saved to the content database. This can make the system heavy loaded after some time.

To load the workbook to SharePoint, choose to "Save & Send" in the Excel Ribbon or just upload the workbook to the site. You should use the PowerPivot Gallery at SharePoint to keep the PowerPivot workbooks.

At the SharePoint server side, you must make sure that relevant features and system services are installed in order to use PowerPivot. To be more specific, Excel Web Access, Excel Calculation Services, PowerPivot System Services, and SQL Server Analysis Services are involved when users interact with published PowerPivot workbook at PowerPivot gallery. Here one can have the same "slicing-and-dicing" , "soring and filtering" user experiences as the PowerPivot for Excel).    As illustrated in the example below,  one can still work with the “slicing-and-dicing” feature inside the web page open by Internet Explorer.

The PowerPivot Gallery is designed for end-users of Self-service BI to use the PowerPivot. So any user of the workbook, with provided access, can re-use the data and even do further analysis based on it. One can click on a specific sheet in the PowerPivot workbook to further analyze the data with given features from Excel Services.

The PowerPivot Gallery has three standard views, i.e., Gallery View, Theatre view, and Carousel view.  The following picture shows the differences of these views. 

After a PowerPivot workbook is published at SharePont site, users of the published workbook can create their own reports based on the PowerPivot workbook data (meaning that the workbook can be used like a data source). When you hover on the icons of the workbooks, you can see a option to create new report. Then there are two options available, Open New Excel Workbook and Create Report Builder Report.  Below is an example.

When you create an Excel Workbook by using the PowerPivot workbook data, the data is provided from the PowerPivot workbook (in SharePoint) in the same ways as provided from an Analysis Service database.

When using Report Builder to create reports based on PowerPivot workbook data, the data is treated as a data connection just like data connections.

Another important work when sharing your PowerPivot report in SharePoint is to consider refreshing data at a PowerPivot workbook stored in SharePoint. You can easily manage the data refresh in the PowerPivot Gallery. The PowerPivot data refresh feature currently does not support data refresh within a day (shame, isn't it). But you can perform an "immediate" data refresh by manually initiate a data refresh command.  And you can also configure the SharePoint to send you an email notification after data has been refreshed.  One thing to remember, if you use linked table in PowerPivot workbook, it will not have any data refresh. Below is an example page of data refresh configuration.

6. Conclusion

This Tutorial is made to describe the key features in PowerPivot in simple words. Apparently one can look into the Microsoft site or Youtube to look for more insights or "live-actions" on these features. 

Hopefully this tutorial can be help for someone looking to start using PowerPivot. Good luck and enjoy the world of Self-service Business Intelligence!

A Simple Tutorial on PowerPivot (3)

 4. Analyze with PowerPivot

The key feature that makes Excel an ideal tool for business analysis is the PivotTable and PivotChart capabilities. Now with Excel 2010, the “slicer” capability (as illustrated below) is also added so that the interactivity is further improved.

PowerPivot provides a set of options to create PivotTable and PivotChart (as illustrated below). Details about each option is out of scope of this tutorial. It is very similar to Excel with more enhancement on slicer and “measures” which can be created by DAX.

One important option, is to create “Flattened PivotTable”. One usage of this option is to print out the data. Another way is to convert the result PivotTable to Excel Formulas so that one can apply Excel functions on the result table.  The option is illustrated below.

It is important to note that the “Field List” for PivotTable or PivotChart under PowerPivot workbooks is different from what it was for other data in Excel. The following picture is an example. The left one is a typical PivotTable Field List while the right one is what you can see in the PowerPivot Field List.

Another interesting feature in PowerPivot is Automatic Relationship Detection. As illustrated below, PowerPivot is being active to detect any potential relationships among tables and fields.

As the core part of data analysis, defining and using measures if always compulsory. PowerPivot provides full support through DAX expressions. Besides the implicit measures which are created while dropping things into the Value area in Pivot chart or Pivot table design. One can also create explicit measures by explicitly specify the DAX expression. To begin creating a measure, right-click on a table name in the PowerPivot field list and select Add New Measure (as illustrated below).

And defining DAX measures requires to use the DAX functions. As illustrated below, the measures are defined in a similar way to Excel formulas.

What to be careful with here is that you should not thing in “rows” when measure values are defined here. And this measure must be defined with an aggregation function (such as sum, count, average, etc.) that will return a single value. In such sense, this is a bit different from the definitions of some calculated columns. This is how it looks like after this measure is defined on the PivotTable.

One more exciting feature to mention, Excel 2010 allows users to create “Named Sets” when using PivotTables (illustrated below). This link provides more detail about this feature: http://blogs.office.com/b/microsoft-excel/archive/2009/10/05/pivottable-named-sets-in-excel-2010.aspx

PowerPivot can also work with Named Sets in Excel.

A Simple Tutorial on PowerPivot (2)

3. Enriching the Data

After importing data, the next natural thing to do is to integrate and enrich the data. There are following ways to enrich data.

First, it is possible to change the actual data types for imported columns. As illustrated below, after choosing a column in a PowerPivot table, one can choose the “Data Type” option in the ribbon to convert the data types.

Compared to changing data type of columns, a second approach is to just change the way data looks like in the table. That is done by re-“formatting” the data which only changes the “view” of data without converting the underlying data type. Formatting changes in PowerPivot are applied to an entire column. This is illustrated below. 

One of the key features of PowerPivot is to create relationships among tables. As illustrated below, it is possible to set up relationships after the PowerPivot tables are imported. The “Create Relationship” option at the “Design” ribbon in PowerPivot window allows one to set up relationship between tables. Due to the limitation of the current version, relationships can only be defined on single columns (as shown below). This means that if you want to establish two distinct relationships between two tables, you must import another instance of the table. It is a bit similar to using SQL to define join statements that use multiple columns in the same table. So users cannot expect to define complicated relationships in PowerPivot. The “Manage Relationship” button shows all the defined relationships of all the imported PowerPivot tables. 

A very existing feature in PowerPivot is that one can define ”measures” (like in Cubes) using the DAX formulas. This includes calculated columns (which are materialized into the table) and measures explicitly defined (which are kept virtual until called). Below is an example of a defined calculated column in a PowerPivot table. The DAX formula in the Fx textbox seems very similar to Excel. But the main difference is that this definition covers a whole column “TotalSales” instead of a single cell in Excel. 

A little bit more about DAX. DAX in fact has a total of 135 functions to help with data analysis. About 80 functions are supported by Excel. DAX also provides functions that operate on tables.The DAX functions can be categorized into eight categories, i.e., Date and Time, Information, Logical, Math and Trigonometric, Statistical, Text, Filter, Time Intelligence. Similar to math and trigonometric functions , only a subset of Excel’s statistical functions is supported by DAX in the current release of PowerPivot. When typing DAX formulas, PowerPivot provides the “autocomplete” feature. One needs to press the Tab key to use this feature.

Like Excel, PowerPivot window provides the ”Sorting” and ”Filtering” options at the “Sort and Filter” Section, and the “Freeze” option at the “View” Section for users to refine the data. This is illustrated below. The column width can also be adjusted in a similar way to Excel.

Another useful feature for improving the view is to use ”hide” and ”unhide” options. As illustrated below, it is possible to hide columns from PowerPivot or PivotTable or both.

After data is loaded to PowerPivot, the “Refresh” button can be used to retrieve the latest source data to PowerPivot. As illustrated below, it is possible to refresh the current table or refresh all tables in PowerPivot. When clicking the “refresh” button, what happens behind the scene is that PowerPivot sends two commands to the VertiPaq engine when you click Refresh:

• The first command is to “Alter”, i.e., to make any metadata changes you have made in the PowerPivot Window.

• The Second command is to “Process”, i.e., to refresh the data from external data sources and any calculations that depend on the refreshed data.

A Simple Tutorial on PowerPivot (1)

1. Introduction

Using PowerPivot is pretty easy. In general, four steps are involved. 

• First, you need to get the data to PowerPivot. This is about importing and doing a little preparation on the data. Data can be brought from relational databases, Analysis Service databases, PowerPivot that have been published at SharePoint sites, Excel and Text files, data feeds (ops, something new from SharePoint), through Copy-Paste actions, linked worksheet, etc.

• Second, after taking everything into PowerPivot, you need to enrich the data through various ways, such as filtering some rows, hide/unhide some columns, formatting columns (or change data types), create calculated columns through DAX definitions, and most important of all, defining relationships among different PowerPivot tables.

• Third, with all data ready to use, it is the time to analyze the data.Here is the time to use DAX to create measures and more Excel functionalities to find out interesting results.

• Forth, and the most important of all, create and share your reports.This is about showing and sharing your reports through SharePoint.

This tutorial has four parts where each part introduces one of the above steps.

Let’s start with the first step.  

2. Data Importing and Preparation

To bring data to PowerPivot is similar to bringing data to SQL Server Express. You will have to be able to connect to different data sources. PowerPivot requires database providers and, in some cases, client libraries in order to be able to import data from these sources. In most cases, the database providers and client libraries are not provided by PowerPivot for Excel and must be acquired and installed separately.

To import data, the simplest way is to import using Table Import Wizard at the PowerPivot window. There are two options when importing data using the Table Import Wizard (so you should open PowerPivot window and click “Table Import Wizard”.

1. Select from a list of tables or view

2. Specify a SQL Query. 

If you choose “Select from a list of tables or view”, the following picture shows the result.

The button “Select Related Tables” gives a big help if you know one of the tables (and have selected it” and wants to make sure that all other relevant tables are included.”

For the selected tables, you can use “Preview&Filter” button to decide whether to include or exclude certain columns or rows from the data. The following picture is an example. 

After selecting all the relevant tables and completing relevant filtering options. Click “Finish” button to start the actual importing process. Result of the Importing process looks like the following.

Now, let’s take a look at the second way, i.e., “Specify a SQL Query” when doing the Table Import Wizard. When choosing this option, “Specify a SQL Query”, one can use the “Design” button to go into the Query Designer to specify the SQL Query. The picture below is an example of the SQL query.

 In addition, Table Import Wizard lets you import data from MS Analysis Services (MSAS) and other PowerPivot workbooks. It’s important to note that the result of MSAS query is typically a "cellset" object, which is a multi-dimensional object. But when the data is imported into PowerPivot, it is transformed into a simple table. When you import data from MSAS you won’t have the same interactivity of slicing and dicing as you typically have when using BI client tools, such as Excel connected to an Analysis Services Unified Dimension Model (UDM) cube. Another very interesting feature of PowerPivot is that  PowerPivot can connect to data in a published PowerPivot workbooks at sharepoint sites. This makes the enterprise mash-ups into reality.  It’s worthy to note that you cannot import data from PowerPivot workbooks if they are .xlsx files on a local disk drive.

Besides importing tables, PowerPivot now supports “data feed”. Data feeds is a new type of technology that provides tabular-like data such that users can subscribe to the data in the same way as the RSS feeds in order to receive updated data as the source changes. Data feeds are based on the Open Data Protocol, which uses the Atom Publishing Protocol and its Data Services URI and Payload Extensions.  Data feed can come from SQL Server reporting services, SharePoint List and any generic data feed that supports the OData protocol.

Here is an example when you choose to import “From Data Feeds” in PowerPivot. PowerPivot understood the URL description and import the initial data from the source.

Text Files is another important source of data. Just like importing data in DTS, PowerPivot works with both Delimited text files and fixed-length-field text files. But to work with the fix-length-field files is a little bit tricky. One of the key step here is to define a schema.ini (exact name as this) at the same location as the file.  The contents of the schema.ini file are described in the MSDN topic at  this link

You can also import from Excel files. This is demonstrated below.

There are also other ways to bring data into PowerPivot. First one is to paste data from clipboard (well, you need to copy these tabular data first), as demonstrated below.

Second one is to create ”Linked Excel tables” which keeps the source and target in sync automatically. The following is an example.

And the result of the linked table is shown in the PowerPivot window with a special icon (the one next to the “DimStore” name in the below picture).

 After specifying the import criteria, PowerPivot will start the importing with the progress reports. It is important to note here that although PowerPivot can import data from many diverse data sources. The data types supported in PowerPivot is very limited compared to the big variety of data types at different sources. One of the things to be alerted is when you are importing data under different currency types, PowerPivot will import currency types as general currency types (“general” to the regional format defined by the machine where PowerPivot is running on). This can definitely lead to unexpected results. But you can use the  PowerPivot formatting tools to adjust the columns’ data type, and format after the import is completed.

The Business Case of Self-service BI

Why Self-service BI makes a difference? If we look at the state of BI before, then the answer is quite clear.

• Only trained and highly-paid BI developers can be used to create BI applications, which is highly customized (which means cost of maintenance, further development, and even decommissioning).

• After BI application is built, it becomes difficult (money, time) to implement changes even the business situations and requirements are changed.

• After having 5 different BI applications, you have to have people with specific skills and knowledge to the ground level of these applications. Each system works in its own way. (so what's the cost of maintenance and operating)
• Since it costs too much, many business analysts start to copy data from production systems and make their own analysis from local PCs. Spreadmarts become popular. Everyone becomes a cowboy and key business decisions are made from some charts in a spreadsheet saved at some one's local PC.
• When people are trying to share the spreadsheet files, the whole business is having the risk of losing confidential data without being aware of it.

Given these scenarios, starting a self-service BI can definitely eliminate the above scenarios by bringing lower cost, shorter time-to-market, reduced IT investment, more precise requirement specifications for corporate BI solutions, and more efficient collaborations among the business users.

To be a bit more specific, here is a few points demonstrating the benefits claimed above.

First, most self-service BI tools start by extending Excel or providing a very easy-to-start interface so that the business users always find it simple and quick to start producing reports, sometimes just like what they have been doing with Excel. And self-service BI means the business users have to serve themselves to produce the end result. There is almost no intervention from IT side, not to mention any "waiting time," "resource bottleneck," etc. This dramatically reduce the development cost for many BI solutions at the business team.

Second, since the development of the BI solutions is at the hand of business users, there is absolutely no communication detour between business-IT-business in order to get most work done. Shorter time-to-market  is ensured since the one who needs it is the one who builds it. Who else to complain about it?

Third, the role of IT in this scenario is to monitor the business work and observe the need for creating corporate BI solutions (which is more reliable in terms of maintenance and operation). What's more important is that after doing these many self-service BI solutions, the business is actually making a very excellent prototype for the potential BI solutions that can benefit more. This saved quite much time of requirement analysis, interviews, escalations and much effort made by the solution architects at every project.

Forth, the concept, which is also a key in many self-service BI solution, is that self-service BI must include a collaborative effort to ensure the communications and reusability among the business team and the IT team. This avoids the spreadmart scenario and in fact, provides a chance to maximize the benefit of solutions developed by the power users at the business team.

Oh, one last point to make. What happens to the corporate BI solutions when self-service BI becomes so popular. The answer is, "nothing." We still need corporate BI solution and we still need to develop these solutions through IT and business corporations. But, the difference is, we've got a better overview of what we need and what we want this time ;)

Basics for Understanding PowerPivot

I often met the questions about PowerPivot these days. "What is it?" "Why should we have it?" "Can PowerPivot do this for me?" 

Well, the best way is to make a list of most basic descriptions of PowerPivot and share with those who need it. So here they are.

• What is PowerPivot

PowerPivot is a new data analysis technology in a way that it makes the "impossible" usage of EXCEL (in the past) to be possible now. So business analysts do not always have to wait for a few months to see the end-result of IT implementation. They can instead do self-service and make many of the BI solution implementations on their own (self-service).

• Self-service BI compared to corporate BI

PowerPivot  (together with Excel, and most importantly, SharePoint) enables the self-service BI capabilities of a company. But this does not mean the era of Corporate BI is over. Instead, corporate BI will have an even more important status since the business is now able to specify more clearly on their requirements and is even able to provide a prototype plus data and results that can be used to validate the IT implementations.

• PowerPivot is not a standalone tool

PowerPivot integrates with Excel (2010 since it needs the size extensions enabled from this version of Excel) and SharePoint 2010.

• Data Analysis Expressions (DAX)

This is one of the key extensions of PowerPivot (in addition to the in-memory analysis engine and the compression algorithms). DAX is a collection of formula elements, i.e., functions, operators and constants that can be used to make expressions or do calculations. It is also available to the Tabular Model projects.

• Managed BI collaboration environment

Clearly PowerPivot has enhanced the possibilities of spread marts and too many standalone worksheets across an enterprise. That's why it is critical to establish a managed BI collaboration environment before promoting the self-service BI concept. Here exists lots of process, rule, training, and SharePointing work. With a SharePoint site, the business can clearly do the self-service work while the IT department puts on certain audit-trails and analysis to find out the need for optimizations, further maintenance and even development of Corporate BI solutions.

• Technical specifications

There is no doubt that PowerPivot includes a good in-memory database. Here is the technical specification that one should always remember. "....The maximum file size of a PowerPivot workbook is 2 GB. There are no restrictions on the amount of data users can import into a workbook, but workbooks exceeding the maximum file size can’t be saved. A 2-GB workbook typically corresponds to a 4-GB dataset, considering a 2:1 compression ratio.... Performance tests show that PowerPivot for Excel can load more than 100 million rows and maintain adequate processing performance with 2 GB of memory. However, test results vary depending

on the compressibility of the data. For fastest processing performance, Microsoft recommends multi-core processors and more than 4 GB of RAM...." (this is from the document "Microsoft SQL Server PowerPivot Planning and Deployment").

• Key elements in PowerPivot for SharePoint

The Timer service (automatically refresh the data), the PowerPivot web services and the PowerPivot Management Dashboard.

• The best picture that shows the roles in BI at enterprises

Look at this one  (or use the following URL http://i.technet.microsoft.com/dynimg/IC396763.jpg)

• The real key to success

Having a focused effort to delivery training, technical documentation and support to the "Power users," and business users is the real critical factor to the success of self-service BI. A good practice is to making a good knowledge base for common questions and training materials.

• Understand the performance gain and loss

What PowerPivot provides a a great performance if everything is inside your memory. To initially get the data from somewhere, there is a potential bottleneck on the network bandwidth. In that case, do not blame PowerPivot but try to decentralize the data just like reducing the heavy traffic around the central station of your city.

• Security and access

PowerPivot is definitely bring challenges to those who work with data access and security control . As a "fat-client" layer in an enterprise landscape, it is very important to identify the security needs and utilize SharePoint as a mitigation layer.