In our superbly data-obsessed world, information is the lifeblood of successful businesses. Therein, the challenge lies in harnessing data's true potential, which has historically been a challenging endeavor for many companies. 

Enter data democratization: a revolutionary concept that has the power to transform organizations, streamline decision-making processes, and unlock innovation. 

And so, today, we'll explore what data democratization means, delve into real-life examples of the problems that arise without it, and highlight its importance across different kinds and sizes of companies. 

## What is Data Democratization?

Data democratization is the process of making data accessible to all relevant stakeholders across an organization, regardless of their technical expertise. 

Traditionally, data was restricted to specialized teams or individuals, leading to bottlenecks, silos, and slower decision-making. Data democratization breaks down these barriers by empowering employees at all levels to access, interpret, and leverage data in their daily operations. 

By promoting a culture of data sharing and collaboration, companies can transform data from a mere resource to a strategic asset.

## …With*out* Data Democratization:

Without data democratization, companies face several challenges that can hinder their growth and competitiveness, such as:

* **Lack of insights:** siloed data inhibits the ability to gain comprehensive insights into business operations. Important correlations and trends may be overlooked, leading to missed opportunities.
* **Sluggish decision-making:** relying on a select few to analyze and interpret data slows down the decision-making process. Real-time data is crucial for making agile and informed decisions.
* **Innovation blockade:** when data is limited to a select group, innovative ideas are stifled. The lack of diverse perspectives and cross-functional collaboration hinders the emergence of breakthrough solutions.

## …*With* Data Democratization

Conversely, companies that embrace data democratization experience a multitude of benefits, including (but not limited to):

* **Faster Decision-Making**: with [business intelligence](https://cube.dev/blog/building-business-intelligence-dashboards-your-team-will-actually-want-to) and general data readily available to all employees, decisions can be made swiftly and based on up-to-date information.
* **Enhanced Productivity**: empowering employees to access and analyze data directly reduces their dependency on data specialists, freeing up valuable time and resources.
* **Informed Innovation**: when every team member has access to data, it fosters a culture of innovation, where creative solutions can emerge from unexpected places.
* **Improved Customer Experience**: by enabling front-line employees to access customer data, companies can deliver more personalized and tailored experiences, leading to increased customer satisfaction.
* **Better Strategic Planning**: data-driven insights are essential for formulating effective strategies and adapting to dynamic market conditions.

## Actual applications of the abstract concept

There _are_ studies, too; for instance, a [McKinsey study](https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-need-to-lead-in-data-and-analytics) found that 64% of respondents from high-performing organizations reported that “data is accessible across the organization,” as opposed to 33% from low-performing organizations. But let's dig into some practical benefits of this otherwise fluffy-sounding concept.

### 1. Empowering Employees with Self-Service Analytics

Data democratization allows employees from various departments, including marketing, sales, finance, and operations, to access and explore data on their own. [Self-service analytics](https://cube.dev/blog/the-future-of-self-service-analytics-data-driven-decisions-meet) tools provide an intuitive interface that enables non-technical users to create reports, perform analyses, and generate insights without relying on data specialists or IT teams. This empowers employees to make data-driven decisions in real-time, reducing the dependency on a limited group of data experts and accelerating problem-solving.

### 2. Enhanced Collaboration and Cross-Functional Insights

In traditional data setups, data is often siloed within departments, hindering collaboration and limiting access to valuable insights. With data democratization, employees from different functions can access a unified data repository, fostering cross-functional collaboration. This collaboration enables a more comprehensive understanding of business operations and allows teams to identify dependencies, correlations, and opportunities that might have been missed in isolated data sets.

### 3. Better Customer Understanding and Personalization

Understanding customers and delivering personalized experiences is crucial for businesses in today's competitive landscape. Data democratization enables customer-facing teams, such as sales and customer support, to access customer data, transaction history, and preferences. Armed with this knowledge, they can tailor interactions, anticipate needs, and provide personalized solutions, leading to improved customer satisfaction and loyalty.

### 4. Real-time Decision Making

In fast-paced business environments, the ability to make decisions quickly is essential. Data democratization enables access to real-time data, which is vital for making agile decisions based on the latest market trends, customer behavior, or operational performance. Real-time insights allow companies to respond promptly to changing conditions and gain a competitive edge.

### 5. Identifying New Business Opportunities

Data democratization facilitates a culture of innovation and empowers employees to explore data creatively. By encouraging every employee to analyze data, companies can surface new ideas and identify potential business opportunities that might otherwise remain hidden. A diverse set of perspectives and innovative ideas can emerge, driving growth and setting the stage for future success.

### 6. Improved Data Quality and Governance

Data democratization does not mean a free-for-all approach to data access. Rather, it promotes responsible data usage through proper governance and data quality standards. By establishing clear data governance policies and roles, companies can ensure data accuracy, consistency, and security. This, in turn, enhances data trustworthiness and safeguards against erroneous decisions based on inaccurate information.

### 7. Competitive Advantage and Business Growth

Data-driven decision-making is a key differentiator in today's market. Companies that embrace data democratization can quickly turn insights into action, making them more adaptable and agile. This ability to respond rapidly to market changes and customer needs gives them a competitive advantage, propelling business growth and expansion.

### 8. Enhancing Data Literacy and Skills

Data democratization fosters a data-driven culture within organizations, leading to increased data literacy among employees. As more individuals engage with data and analytics tools, their analytical skills improve, driving a shift towards a data-centric mindset. This upskilling of the workforce ensures that data-driven insights become an integral part of daily operations.

### 9. Optimizing Resource Allocation

With data democratization, companies can optimize resource allocation by identifying inefficiencies and areas of improvement. Data insights enable organizations to allocate their budgets, time, and workforce more strategically, ensuring that resources are directed where they can have the most significant impact.

## Data Democratization: important across the board

Data democratization is not limited to any particular industry or company size. Its significance spans across various sectors, including startups, small and medium-sized enterprises (SMEs), and large enterprises.

In the early stages, startups require data to validate their assumptions, identify target markets, and refine their business models. Democratizing data from the outset empowers them to make crucial decisions that can determine their success.

Next up: SMBs. Installing a data-driven culture enabled by data accessibility, smaller businesses can level the playing field and to compete with larger rivals on customer understanding.

And, in large, complex organizations, data is often scattered across multiple departments. Data democratization unifies the organization, ensuring everyone has access to the same information, leading to cohesive strategies and streamlined operations.

## The Role of the Semantic Layer

To effectively implement data democratization, companies can hugely leverage a universal semantic layer, which acts as a bridge between the raw data and end-users. 

A universal semantic layer provides a unified view of the data, abstracting its complexities and making it easily accessible to all employees, and thereby accomplishes several key tasks in installing data accessibility:

1. **Data Abstraction**: the semantic layer hides the technical details and complexities of the underlying data sources, enabling non-technical users to interact with the data effortlessly.
2. **Data Governance**: by defining clear data access permissions and policies in a centralized manner, immediately downstream of data sources, the semantic layer ensures that sensitive information is only accessible to authorized users. This cohesive governance preserves data security and compliance.
3. **Self-Service Analytics**: With a semantic layer in place, employees can use intuitive tools and interfaces to explore and analyze data without requiring specialized technical skills. Companies can use [a myriad of tools](https://cube.dev/blog/the-critical-components-an-example-embedded-analytics-tools-stack), from out-of-the-box BI and notebooks to super customizable front-end libraries, in order to give employees a precisely tailored data experience. 
4. **Consistency and Accuracy**: The semantic layer ensures consistent data definitions and terminology throughout the organization, by defining metrics upstream of every data application in the company’s stack. Doing so eliminates cross-collaborative confusion and errors resulting from inconsistent data interpretations and keeps everyone on the same page.
5. **Time-to-value**: when everything data-orchestration is taken care of once, upstream of every data application, it makes it a lot easier (and just generally feasible) for teams to build the right data experiences for them.  

## To sum it up

Data democratization is not just a buzzword; it's a transformative approach that empowers companies to unleash their true potential. By breaking down data silos and fostering a data-sharing culture, companies can make more informed decisions, foster innovation, and deliver better customer experiences. 

And, leveraging a semantic layer as part of this democratization process enhances accessibility, data governance, and overall data-driven decision-making, enabling companies to thrive in a data-driven world. Embrace the power of data democratization and witness the remarkable transformation it can bring to your company.

Curious how a semantic layer can empower your organization to move towards data democratization? [Reach out to us](https://cube.dev/contact)—we’d love to talk :) 


Tamar Barlev

Best Practices

Or, how to unleash your company's potential.

The Importance of Data Democratization

Business Intelligence (BI) dashboards are powerful tools that enable organizations to make informed decisions by presenting data in a visually appealing and easily understandable manner. 

However, building BI dashboards for internal stakeholders that drive adoption and return on investment—ergo, those that they’ll actually _want_ to use—requires careful consideration of usability and user experience (UX). 

So today, we’ll explore the best practices that can help create effective business intelligence dashboards that maximize user engagement and lead to tangible business outcomes.

## Why it’s important to build business intelligence dashboards (your team will actually want to use)

Building BI dashboards that your team will actually want to use is important for _many_ reasons; evidently, your company would be investing resources in doing so for no reason. But let’s take a look at a few, shall we?

### 1. Empowering Informed Decision-Making

An engaging BI dashboard puts valuable data at stakeholders' fingertips, empowering them to make well-informed decisions swiftly. By presenting insights in a clear and visually appealing manner, stakeholders can quickly identify trends, spot opportunities, and respond to challenges effectively.

### 2. Enhancing Productivity and Efficiency

BI dashboards that are easy to use and provide real-time data reduce the time spent on data analysis and reporting. Internal stakeholders can focus on strategic initiatives, driving productivity, and efficiency across departments.

### 3. Fostering Data-Driven Culture

When stakeholders actively use BI dashboards, they embrace a data-driven culture within the organization. Data-backed decisions become the norm, promoting a collaborative environment where everyone leverages insights to drive growth and innovation.

### 4. Identifying Untapped Opportunities

An engaging BI dashboard helps uncover hidden opportunities and potential areas for improvement. Stakeholders can pinpoint underperforming aspects of the business, leading to proactive interventions and optimizing resources.

### 5. Gaining a Competitive Edge

Organizations that prioritize UX in their BI dashboards gain a competitive edge in the market. The ability to quickly access and interpret data gives them an agility advantage, enabling them to respond swiftly to market changes and customer demands.

### 6. Ensuring Return on Investment (ROI)

BI dashboards are significant investments for organizations. To ensure a meaningful ROI, it is crucial that stakeholders fully utilize the dashboard's capabilities. High adoption rates translate into greater returns as insights lead to better decision-making, cost savings, and revenue growth.

### 7. Promoting Collaborative Decision-Making

A user-friendly BI dashboard encourages collaboration among different teams and departments. With the ability to share insights and discuss data-driven strategies, stakeholders can align their efforts toward common goals. 

## How to build better business intelligence dashboards

### 1. Know Your Audience

The foundation of a successful BI dashboard lies in understanding the needs, preferences, and pain points of your internal stakeholders. Conduct thorough user research and engage in conversations with key stakeholders to identify their specific requirements. By knowing your audience, you can tailor the dashboard's design and content to meet their expectations, ultimately leading to higher adoption rates.

### 2. Define Clear Objectives

Before diving into dashboard development, establish clear objectives and key performance indicators (KPIs) that the BI dashboard is expected to address. Whether it's monitoring sales performance, tracking project progress, or analyzing customer behavior, defining measurable goals will ensure the dashboard's relevance and impact on the organization's overall success.

### 3. Prioritize Data Relevance and Accuracy 

Relevant and accurate data is the lifeblood of any BI dashboard. Ensure that the data sources are trustworthy and regularly updated. Additionally, present the most critical information prominently, prioritizing real-time or near-real-time data, if applicable, to ensure that the dashboard provides timely insights for decision-making.

### 4. Design for Intuitive Navigation

A key aspect of usability is ensuring the dashboard's navigation is intuitive and user-friendly. Arrange data elements logically, provide clear labels and descriptions, and avoid clutter. Implement a clear and consistent hierarchy to guide users seamlessly through the dashboard, allowing them to drill down to the details they need effortlessly.

### 5. Embrace Data Visualization Best Practices

Visual representations of data are powerful tools for comprehension and insight. Utilize data visualization best practices to convey complex information clearly and concisely. Choose appropriate chart types, color schemes, and label placements to facilitate easy interpretation of the data. Avoid using overly complex visuals that may confuse users or obscure key insights.

### 6. Provide Interactivity and Customization Options

Empower internal stakeholders by offering interactive features and customization options within the dashboard. Users should be able to filter, sort, and personalize the dashboard to suit their unique requirements. By allowing users to interact with the data, you enhance engagement and enable them to derive deeper insights.

### 7. Optimize for Responsiveness

In today's fast-paced business environment, stakeholders access business intelligence dashboards across various devices, including desktops, laptops, tablets, and smartphones. Ensure the dashboard's design is responsive and adaptive, so it seamlessly adjusts to different screen sizes and orientations. A mobile-friendly dashboard enables stakeholders to stay informed, even while on the go.

### 8. Monitor and Analyze User Behavior

Keep a close eye on how stakeholders interact with the dashboard. Analyze user behavior to understand which sections are most and least popular, where users drop off, and what actions lead to desired outcomes. Utilize this data to continuously improve the dashboard's usability and performance.

### 9. Foster Collaborative Workflows

business intelligence dashboards often serve as a shared resource for internal stakeholders across different departments. Facilitate collaboration by integrating features that enable users to share insights, export data, or generate custom reports. By fostering collaborative workflows, you encourage cross-functional engagement and promote a data-driven culture within the organization.

### 10. Regularly Update and Improve

A static BI dashboard loses its appeal over time. To sustain high adoption rates and drive ROI, invest in regular updates and improvements. Address user feedback, incorporate new data sources, and stay up-to-date with evolving best practices in BI dashboard design. A living dashboard that evolves with the organization's needs will keep stakeholders engaged and invested in its insights.

## Strategies for building better business intelligence dashboards

### User Persona Profiling:

Conduct thorough user persona profiling to understand the specific needs and pain points of different user groups within the organization. For instance, a sales manager might require real-time updates on key performance metrics, while a marketing analyst might need historical data for trend analysis. Tailor the dashboard's layout and content to cater to the unique requirements of each persona, ensuring high user satisfaction and adoption.

### Goal-Oriented Dashboard Design:

Create goal-oriented dashboards by aligning the key performance indicators (KPIs) with the objectives of individual departments or teams. For example, if the HR team aims to reduce employee turnover, the dashboard should prominently display metrics related to employee satisfaction, turnover rates, and retention strategies.

### Data Storytelling:

Incorporate data storytelling techniques into the dashboard design to make the insights more compelling and understandable for stakeholders. Use narrative elements such as headings, annotations, and callouts to guide users through the data and help them draw meaningful conclusions. An example could be using a storytelling approach to showcase the impact of marketing campaigns on sales figures.

### Drill-Down Functionality:

Implement drill-down functionality that allows users to explore data at different levels of granularity. For instance, a financial analyst should be able to drill down from a high-level overview of company expenses to specific cost categories and individual transactions. This empowers stakeholders to investigate anomalies and gain deeper insights.

### Actionable Alerts and Notifications:

Integrate actionable alerts and notifications into the dashboard to keep stakeholders informed about critical changes or events in real-time. For instance, if inventory levels drop below a certain threshold, the inventory management team should receive an automated alert to take necessary actions promptly.

### Collaborative Annotations:

Enable users to add comments, annotations, or share their insights within the dashboard. This fosters collaboration among teams, allowing them to exchange ideas and collectively arrive at data-driven decisions.

### Personalization Options:

Provide users with the ability to personalize their dashboard by selecting preferred data visualization styles, default filters, or customized layouts. For example, allowing sales representatives to choose their preferred sales metrics and target regions.

### Performance Optimization:

Optimize the dashboard's performance to ensure fast loading times, especially when dealing with large datasets. Implement caching mechanisms and data pre-processing techniques to deliver a smooth user experience.

### A/B Testing:

Employ A/B testing to experiment with different dashboard layouts, visualizations, or interaction designs. Measure user engagement and satisfaction to identify the most effective dashboard design that drives higher adoption and ROI.

### User Training and Support:

Provide comprehensive user training and ongoing support to ensure that stakeholders understand the dashboard's capabilities and use it to its full potential. Conduct workshops, create video tutorials, or offer documentation to aid users in harnessing the dashboard's power for data analysis and decision-making. 

---

## To sum things up.

Building business intelligence dashboards that internal stakeholders genuinely want to use is a critical success factor for any organization. By driving adoption through usability and UX best practices, organizations empower stakeholders with the right data, at the right time, in the right way. This fosters a data-driven culture, enhances productivity, and enables the organization to stay ahead in a competitive landscape. With a strong focus on building engaging and valuable business intelligence dashboards, organizations can unlock the full potential of their data, driving growth, and achieving sustained success.

Want to build better BI dashboards, faster? 

Check out Cube—the semantic layer for every data application. With upstream [data modeling](https://cube.dev/product/data-modeling), [caching](https://cube.dev/product/caching-and-data-performance), and [data access control](https://cube.dev/product/data-access-control), you can ensure your data applications are consistent, fast, and secure. And with its [data API layer](https://cube.dev/product/data-apis), you can curate your stack and tailor your app precisely to your users, without the resource demands of building an application from scratch. 

Building Business Intelligence Dashboards (your team will actually want to use)

As businesses collect more and more data from users, customers, and partners, the need for secure data access becomes ever more significant. 

Think of it like this: if you don’t have tight controls on who has access to your valuable data, then anyone could potentially view or even modify it. Essentially, exactly why security and authentication in a semantic layer are vital to protecting your—or your users’—data.

So, let’s talk about how semantic layers help implement tighter data security. We’ll also explore the importance of multitenancy, authentication tokens, and data access controls in general (both row-based and column-based) as they relate to HIPAA and SOC 2 compliance. 

## What is a Semantic Layer?

Simply put, a semantic layer is a layer of abstraction that sits between the user interface (UI) and the underlying database of a data application. And, IOHO (in our humble opinion), a good semantic layer—or rather, [a complete, universal semantic layer](https://cube.dev/)—should do more than only metrics definitions. 

Its duties encompass data modeling, sure—ensuring that software engineering best practices such as the DRY principle are being employed in metrics orchestration to organize and make data across the stack consistent. 

However, a semantic layer should also include a [caching layer](https://cube.dev/product/caching-and-data-performance) to speed up and application performance and make querying cost-effective. 

Also in the requirements for a good semantic layer? An [API layer](https://cube.dev/product/data-apis) to maintain complete data accessibility across a flexible stack. 

And lastly—but certainly not least…ly—it must provide extra security for sensitive information stored within the database by limiting who can view or modify specific information. 

Essentially, a *good* semantic layer *must* also include [a governance layer or data access controls](https://cube.dev/product/data-access-control). 

## Why is it important to orchestrate data access controls upstream of data apps?

Having data access controls as a part of a semantic layer is crucial for maintaining data security and integrity. It enables organizations to limit access to sensitive data, ensuring only authorized individuals can access the information they need. 

### A fictional example of what *not* to do:

Imagine, if you will, a highly fictional, highly hypothetical credit card score company called Creditlandia, which holds the extremely precious financial data of millions.

Creed is a data engineer at Creditlandia. Moreover, Creed is very tired because he has to manually orchestrate security context for each of Creditlandia's downstream apps; they are all fed data from its massive data stores. 

One day, Creed—the human he is—makes a mistake in the tedious and exhausting work that he does. 

As a result, a begrudged member of the CX team suddenly sees they have read and write access to every customer’s data. They delightedly export this PII (Personal Identifiable Information) to a hard drive, sell it on the dark web, and retire happily at 36. Creed and management, meanwhile, are not so happy. 

And, likely, neither are their millions of customers.

…so, do we see where Creditlandia went wrong? 

By faciliating a data pipeline against best practices; one that necessitates duplicated processes and envitably opens gaps in consistency (of security, metrics, and so on). 

### Orchestrating security context and data governance with a semantic layer

Suppose data access controls are installed in a semantic layer—the middleware sitting downstream of every data source and upstream of every data app. In that case, Creed joyously no longer has to orchestrate security context for each manually. 

An organization can then be rest assured that only the right people have the right access to the right information. An executive can have every kind of access to every customer record, but a customer service rep can only read anonymized versions of their open tickets.  

Orchestrating granular data access controls in a semantic layer also enables organizations to maintain compliance with regulatory requirements. Centralizing security context provides a comprehensive view of data access across the organization. Ergo, companies can easily identify who has access to what data, enabling them to quickly audit and adjust access controls as necessary.

Overall, implementing data access controls upstream of every data application is vital for providing a more efficient means of managing data security across the organization. And in doing so, enhancing governance and ensuring compliance.

Basically, don’t be like Creditlandia. 

### The details about data access controls

#### What are Authentication Tokens?

Authentication tokens are unique identifiers assigned to users upon successful login or authentication. These tokens are then used to authenticate and authorize the same user's access to the system or application, particularly when accessing sensitive data. 

In this way, authentication tokens are a crucial aspect of data access control because they ensure that only authorized and authenticated users can access confidential information or processes within an organization's information system.

#### What is Role-Based Access Control & what is Column-Based Access Control?

[Role-based access control (RBAC)](https://cube.dev/docs/recipes/role-based-access) is a security feature that restricts access within a system by assigning permissions and privileges based on an individual's role or position in the organization. For instance, managers would have permission to access information that entry-level employees would not. This restricts unauthorized personnel from accessing sensitive information and provides an added layer of protection for confidential data.

For example, RBAC might be used within a financial institution. The bank has several departments such as customer service, accounting, and marketing. Each department has different duties and responsibilities, and there is a risk of data breaches if employees can access information outside their scope of work. 

RBAC could be implemented to restrict data access based on employees' job titles or roles, preventing unauthorised personnel from accessing private client data. This enhances security and privacy while ensuring that employees can access the information necessary to perform their duties. 

[Column-based access control (CBAC)](https://cube.dev/docs/recipes/column-based-access) works similarly, but restricts which columns in a table are visible instead of restricting rows. 

For example, an employee may be allowed to see all customer records but may only be able to view certain columns such as first name or last name but not address information or phone number information unless they have additional authorization tokens granting them permission to do so.

### What is Multitenancy?

[Multitenancy](https://cube.dev/docs/config/multitenancy#multitenancy) is when multiple organizations use the same instance of software, rather than each organization having its own. For example, many companies now use cloud software such as Salesforce or Microsoft Office 365, instead of buying a server for their organization's exclusive use. 

Of course, this is highly applicable to [embedded analytics](https://cube.dev/use-cases/embedded-analytics) applications—for example, e-commerce dashboards for small businesses. One vendor—Quirky Hats 4 U—should not have access to the sales information of another, say, Funny Hats 4 Us. 

Multitenancy is essential in terms of security because it allows organizations to share resources while maintaining individualized  protection over their own data sets. In addition, authentication tokens ensure that only authorized users can access specific databases within an instance of multi-tenant software.

## Data Access Controls, HIPAA & SOC 2 Compliance

Data access controls help protect sensitive PII, such as social security numbers or health records, by prohibiting unauthorized users from accessing confidential files without proper authorization tokens. 

These controls are critical when dealing with [HIPAA](https://www.cdc.gov/phlp/publications/topic/hipaa.html#:~:text=The%20Health%20Insurance%20Portability%20and,the%20patient's%20consent%20or%20knowledge.) compliance since healthcare organizations must adhere strictly to HIPAA regulations concerning patient privacy. Failing to do so can result in fines or other penalties imposed by the Department of Health and Human Services Office for Civil Rights (OCR).

Similarly, [SOC 2](https://www.finra.org/compliance-tools/cybersecurity-checklist) compliance requires organizations dealing with financial services, or handling large amounts of sensitive PII data, to adhere strictly to SOC 2 regulations regarding storage and transmission requirements. Failing here, too, can result in hefty fines from regulatory agencies like The Financial Industry Regulatory Authority (FINRA). 

A well-implemented semantic layer can help guarantee compliance with these regulations by providing granular controls over who has access to what pieces of PII at any given time. By doing so, organizations ensure that data remains secure even after being transferred between systems or networks.

## Summing it up: why a semantic layer is key in data security and authentication

All in all, security and authentication in a semantic layer are essential for protecting your valuable business assets—particularly when dealing with PII. 

Not only does this type of setup allow you granular control over who has access to what kinds of data at any given time, but it also ensures your company complies with applicable laws like HIPAA and SOC 2.

If you’re interested in learning more about how crucial semantic layers are in a modern data stack—not just for security but also data consistency, speed, and accessibility—[try Cube for free](https://cubecloud.dev/auth/signup) or [reach out to us](https://cube.dev/contact).

Onward and upwards, 
Tamar 


Or, why you *need* a middleware that includes data access controls.

The (Earnest) Importance of Orchestrating Security Context in a Semantic Layer: HIPAA and SOC 2

If you’re a web developer, I’m sure that you’ve heard about *tree-shaking*. It’s often understood as the removal of unused code from JavaScript bundles. However, do you know that not all code that looks unused is tree-shakeable?

In this blog post, we’ll learn more about tree-shaking and tree-shakable design. You’ll know to write JavaScript code in a tree-shakeable way and produce smaller bundles. If you’re building a JavaScript library, is a must.

> This is a part of the series of posts on [best practices](https://cube.dev/blog/category/best-practices) where I’m sharing bits of my experience contributing to open source. It’s both my hobby and job at [Cube](https://cube.dev) where we create and maintain open source tools for building data applications.

## What is tree-shaking?

Wait! If tree-shaking is the removal of unused code from bundles, then how is it different from “dead code elimination” that has been a feature of JavaScript minifiers for quite a while? It’s a fair question, and the answer is that tree-shaking is a later term that is only relevant if JavaScript bundlers and ES6 imports/exports are used.

Bundlers analyze ES6 imports/exports at build time to find unused code. Since ES6 imports/exports are a part of JavaScript syntax, it’s quite easy to perform a static analysis of used and unused exports.

On the other hand, CommonJS exports are resolved at run time and can be dynamic. That’s why static analysis of used and unused exports is impossible for CommonJS modules.

**Let’s compare how Webpack bundles CommonJS and ES6 modules.** In this example, we export two functions (`sum` and `div`) from a CommonJS module but only one of them (`sum`) is imported and used. (Note that here the code in `bundle.js` is shown *before minification*.)

```javascript
// CommonJS
// module.js

exports.sum = function sum(a, b) {
  return a + b;
};

exports.div = function div(a, b) {
  return a / b;
};

// index.js

const { sum } = require('./module');

console.log(sum);

// bundle.js

(() => {
  var __webpack_modules__ = ({
    739:
      ((__unused_webpack_module, exports) => {
        exports.sum = function sum(a, b) {
          return a + b;
        };

        exports.div = function div(a, b) {
          return a / b;
        };
      })
  });

  function __webpack_require__(moduleId) {
    /* ... */
  }

  (() => {
    const { sum } = __webpack_require__(739);

    console.log(sum);
  })();
})();
```

You can see that Webpack added the `__webpack_require__` function to the bundle to support CommonJS module loading. Also, the `div` function is present in the bundle and won’t be removed after minification because it’s assigned to an object property (`exports.div`) and will be considered “used”. Compare to an ES6 module:

```javascript
// ESM
// module.js

export function sum(a, b) {
  return a + b;
}

export function div(a, b) {
  return a / b;
}

// index.js

import { sum } from './module.js';

console.log(sum);

// bundle.js

(() => {
  "use strict";

  function sum(a, b) {
    return a + b;
  }

  function div(a, b) {
    return a / b;
  }

  console.log(sum)
})();
```

You can see that the `div` function was copied “as is” and is not used in any way. When you run Webpack in production mode, it will remove this function. (Internally, Webpack relies on [Terser](https://terser.org/) for minification; [Rollup](https://rollupjs.org/) performs tree-shaking and dead code elimination on its own.) 

## What is tree-shakable code?

In the example above, two functions were exported from the module and only the “used” one made it to the bundle. What if not only functions are exported? (Note that here the code in `bundle.js` is shown *after minification*.)

```jsx
// module.js

export const foo = 'bar'.toUpperCase();

export const PI_2 = Math.PI * 2;

export function sum(a, b) {
  return a + b;
}

// index.js

import { sum } from './module.js'

console.log(sum);

// bundle.js

(() => {
  "use strict";
  "bar".toUpperCase(), Math.PI;
  console.log((function(a, b) {
    return a + b;
  }));
})();
```

You might be surprised that all exported entities are present in the bundle, although only the `sum` function was imported. Let’s explore why.

### Pure code

Removing unused functions is quite straightforward. If a function is never called, its declaration should be considered “dead code” and removed from the bundle. It works because function declarations have no *side effects*: they don’t update global variables, don’t work with DOM, etc. (Please don’t confuse function declarations with function calls; the latter may have side effects.)

Consider the first exported entity: `'bar'.toUpperCase()`. This is a function call and it may have side effects. Well, we know that `toUpperCase` is a *pure function*, meaning calling it has no side effects. However, minifiers know nothing about the standard library. That’s why the function call wasn't removed from the bundle so the code doesn’t break.

Now consider the second exported entity: `Math.PI * 2`. This is an arithmetic operation (a constant is multiplied by two) and should not have any side effects. Now take a closer look: what does `Math.PI` do? It denotes access to a property of an object and it may have side effects. Access to a variable is always pure, but access to a property may invoke a getter method that can do anything. Minifiers don’t know how pure that code is so they leave it intact. (Obviously, this logic applied to setters as well.)

### How to make pure code tree-shakable?

If you know that some code is pure, there’s a way to tell minifiers about that. By convention, placing the `/* #__PURE__ */` comment before a function call or a constructor call (`new`) marks them as pure. If you need to mark access to a property as pure, you can wrap it with a pure function:

```jsx
// module.js

export const foo = /* #__PURE__ */ 'bar'.toUpperCase();

function getPi2() {
  return Math.PI * 2;
}

export const PI_2 = /* #__PURE__ */ getPi2();

export function sum(a, b) {
  return a + b;
}

// index.js

import { sum } from './module.js'

console.log(sum);

// bundle.js

(() => {
  "use strict";
  console.log((function(a, b) {
    return a + b;
  }));
})();
```

Now it looks better, right?

Here are a few real-world examples of code purification from my contributions to open source:

- In [Chart.js](https://www.chartjs.org), tree-shaking [didn’t work as expected](https://github.com/chartjs/Chart.js/issues/10163) because of assignments to class properties ([replaced with static properties](https://github.com/chartjs/Chart.js/pull/10504/files#diff-20242c6d6e67c730511b603d00054d6c95d7698f180cc1eaeeff4dd3278b9c98)).
- In [react-chartjs-2](https://react-chartjs-2.js.org), some exported entities are constructed by calling a factory method ([marked as pure](https://github.com/reactchartjs/react-chartjs-2/blob/59b64feb7d9a24b3557e253f9089225031ce2b72/src/typedCharts.tsx#L33-L59)).

You can experiment with dead code detection yourself in [Terser REPL](https://try.terser.org/).

### `sideEffects: false`

If you’re building a library, you should know about the `sideEffects` property in the `package.json` file. It's similar to `/* #__PURE__ */` but on a module level instead of a statement level. By setting it to `false`, you can mark the entire code of your library as pure:

```json
{
  "sideEffects": false
}
```

You can also set it to an array of files that contain side effects:

```json
{
  "sideEffects": ["./dist/polyfill.js"]
}
```

You can learn more about the `sideEffects` property in the [Webpack documentation](https://webpack.js.org/guides/tree-shaking/#mark-the-file-as-side-effect-free). Please pay attention to this behavior: a package, marked as pure with `sideEffects: false`, will be skipped entirely if imported as a whole.

```javascript
// module.js

export const foo = 'bar'.toUpperCase();

export const PI_2 = Math.PI * 2;

export function sum(a, b) {
  return a + b;
}

// index.js

import './module.js'

console.log('noop');

// bundle.js

(() => {
  "use strict";
  console.log('noop');
})();
```

However, if individual entities are imported, everything works as described in the previous section and you still need to use `/* #__PURE__ */`:

```javascript
// module.js

export const foo = 'bar'.toUpperCase();

export const PI_2 = Math.PI * 2;

export function sum(a, b) {
  return a + b;
}

// index.js

import { sum } from './module.js'

console.log(sum);

// bundle.js

(() => {
  "use strict";
  "bar".toUpperCase(), Math.PI;
  console.log((function(a, b) {
    return a + b;
  }));
})();
```

Things to consider when using `sideEffects: false`:

- If your library code is already covered with `/* #__PURE__ */`, using `sideEffects: false` will have no effect.
- If your library provides global style sheets (not [CSS modules](https://webpack.js.org/loaders/css-loader/#modules)), they should also be added to the array of files that contain side effects:

```json
{
  "sideEffects": ["./dist/index.css"]
}
```

It’s required because such style sheets are usually imported directly:

```javascript
import 'mylib/dist/index.css'
```

- Beware when developing a library: bundlers will look for `sideEffects: false` not only in the dependencies of your library but in its `package.json` as well.

## Tree-shakable design

Now we know how tree-shaking works in principle. However, we should also design a library in a tree-shakeable way. For instance, if the entire library is packed in a single class, tree-shaking wouldn’t be able to remove any code.

Here are the rules of tree-shakable design:

1. All entities, including class methods, that might not be used should be exported separately (so they can potentially be removed from the bundle).
2. Classes and objects should be exported cautiously (because their contents can’t be removed from the bundle).
3. All code should be free of side effects by default. Minimal, if any, instances of non-pure code should be permitted. 

Consider the following examples.

### Example A

Imagine we’re writing an internationalization library:

```javascript
// non-tree-shakable

import I18n from 'my-i18n-lib';

const i18n = new I18n(config);

console.log(i18n.__('foo'));
// console.log(i18n.__n('bar', 5));

// tree-shakable

import { I18n, translate, translatePlural } from 'my-i18n-lib';

const i18n = new I18n(config);
const __ = i18n.bind(translate);

console.log(__('foo'));
```

🙅 In the non-tree-shakable variant, a single giant class is exported. The bundle will include all class methods, including unused ones.

🙆 In the tree-shakable variant, potentially unused class methods are exported separately as functions. The bundle will include only used functions.

### Example B

Imagine we’re writing a data visualization library:

```javascript
// non-tree-shakable

import { BarChart } from 'my-charting-lib';

new BarChart('#chart', data, {
  axisX: {
    type: 'auto'
  },
  axisY: {
    type: 'fixed' // 'step'
  }
});

// tree-shakable

import {
  BarChart,
  AutoScaleAxis,
  FixedScaleAxis,
  StepAxis
} from 'my-charting-lib';

new BarChart('#chart', data, {
  axisX: {
    type: AutoScaleAxis
  },
  axisY: {
    type: FixedScaleAxis
  }
});
```

🙅 In the non-tree-shakable variant, all axis types should be imported inside `BarChart` so it can use some of them according to the options provided. The bundle will include all classes for all axis types.

🙆 In the tree-shakable variant, only necessary axis types are imported and provided to `BarChart`. The bundle will include classes only for used axis types.

(Full disclosure: this is a real-world design decision from [Chartist](https://github.com/chartist-js/chartist), an open-source data visualization library I maintain and contribute to.)

### Example C

Imagine we’re writing a more complex data visualization library like [Chart.js](https://github.com/chartjs/Chart.js). In that case, you may choose to implement [dependency injection](https://medium.com/geekculture/dependency-injection-in-javascript-2d2e4ad9df49):

```javascript
import {
  Chart,
  CategoryScale,
  LinearScale,
  PointElement,
  LineElement,
  Title,
  Legend
} from 'chart.js';

Chart.register(
  CategoryScale,
  LinearScale,
  PointElement,
  LineElement,
  Title,
  Legend
);

new Chart(ctx, {
  type: 'line',
  data: data,
  options: {
    plugins: {
      legend: {
        position: 'top',
      },
      title: {
        display: true,
        text: 'Chart.js Line Chart',
      },
    }
  }
});
```

🙆 In this tree-shakable implementation, dependency injection via `Chart.register` is used. Library code is very well split into individual parts that are exported separately. The bundle will include classes only for used entities.

## Keeping code tree-shakable

A one-time effort to write or rewrite a JavaScript library in a tree-shakable way is not enough. One day, due to lousy code or code review, tree-shaking might brake.

However, you can use the [Size Limit](https://github.com/ai/size-limit) library to control the size of a bundle or individual exports. Here’s an example configuration:  

```javascript
// .size-limit.json

[
  {
    "path": "dist/chart.esm.js",
    "limit": "19 KB",
    "import": "{ BarController, BubbleController, DoughnutController, LineController, PolarAreaController, PieController, RadarController, ScatterController }"
  },
  {
    "path": "dist/chart.esm.js",
    "limit": "14 KB",
    "import": "{ ArcElement, LineElement, PointElement, BarElement }"
  },
  {
    "path": "dist/chart.esm.js",
    "limit": "27 KB",
    "import": "{ Decimation, Filler, Legend, SubTitle, Title, Tooltip }"
  },
  {
    "path": "dist/chart.esm.js",
    "limit": "22 KB",
    "import": "{ CategoryScale, LinearScale, LogarithmicScale, RadialLinearScale, TimeScale, TimeSeriesScale }"
  }
]
```

Size Limit integrates with CI such as GitHub Actions and will alert you if any pull request drastically increases the bundle size, possibly due to broken tree-shaking.

## What’s next?

Please use tree-shaking in your code at all times. If you’d like to learn how tree-shaking works by example, check out [this sandbox](https://codesandbox.io/s/webpack-tree-shaking-test-glb8ov?file=/src/module.js) I’ve built.

If you have an existing application or library:

- make sure it complies with the three rules of tree-shakable design;
- make sure pure functions are marked as such with `/* #__PURE__ */`;
- make sure [Size Limit](https://github.com/ai/size-limit) is used to notify if tree-shaking brakes.

Good luck!

Dan Onoshko

How to build tree-shakeable JavaScript libraries

If you’re a web developer, I’m sure that you use bundlers (e.g., [Webpack](https://webpack.js.org), [Rollup](https://rollupjs.org/guide/en/), or [Parcel](https://parceljs.org)) which transpile the JavaScript code of your application with [Babel](https://babeljs.io) under the hood. No doubt that you also use various dependencies to cut off the development time.

However, I rarely see developers transpiling the code of their dependencies because everything seems to work fine without it, right? Wrong! Here’s why...

> This is a part of the series of posts on [best practices](https://cube.dev/blog/category/best-practices) where I’m sharing bits of my experience contributing to open source. It’s both my hobby and job at [Cube](https://cube.dev) where we create and maintain open source tools for building data applications.

## Adoption of ESM

Before browsers and Node.js got native support for ES modules, an npm package could contain several variants of source code:

- [CommonJS](https://en.wikipedia.org/wiki/CommonJS) variant that doesn’t use [modern features](http://es6-features.org/#ExpressionBodies) of JavaScript such as arrow functions. It’s compatible with most versions of Node.js and browsers. The source code location is specified in the `main` field of `package.json`.
- module variant that has ES6 imports/exports but still doesn’t use modern features of JavaScript. Modules allow for [tree-shaking](https://developer.mozilla.org/en-US/docs/Glossary/Tree_shaking), i.e., excluding unused code from a bundle. The source code location is specified in the `module` field of `package.json` (see [this discussion](https://stackoverflow.com/questions/42708484/what-is-the-module-package-json-field-for) for details).

Obviously, not every npm package follows this convention. It’s a choice that every author of a library makes on their own. However, it’s quite common for browser-side and universal libraries to be distributed in two variants. 

[Web browsers](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) have natively supported ES modules for more than three years already, and [Node.js](https://nodejs.org/api/esm.html) supports them since version 12.20 (released in November 2020). That’s why authors of libraries now include one more variant of source code for execution environments that natively support ES modules, and many packages have completely removed the support for CommonJS.

## Perks and features of ESM

It’s important to understand that native ES modules are very much different than modules that have ES6 imports/exports. Here are a few reasons:

- The way we are used to using imports/exports will not work natively. Such code is intended for further processing by a bundler or a transpiler.
- Native ES modules don’t support `index` file name and extension resolution, so you have to specify them explicitly in the import path:

    ```javascript
    // Won't work:
    import _ from './utils'

    // Works:
    import _ from './utils/index.js'
    ```

- If an npm package has ES modules, you have to add `"type": "module"` to `package.json` and specify the source code location in the `exports` field (see [docs](https://nodejs.org/api/packages.html#exports) for details).

You can check [this blog post](https://2ality.com/2019/10/hybrid-npm-packages.html) by Axel Rauschmayer to learn more about ES modules.

## Support for JavaScript features

Look! Since we know which versions of web browsers and Node.js support ES modules, we also know for sure which features of JavaScript we can use in the source code.

For instance, all web browsers that support ES modules natively also support [arrow functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions), so we don’t need to avoid using them or use Babel to transpile them to regular functions. Authors of libraries know that and ship the source code that leverages all modern features of JavaScript.

## Transpilation of dependencies’ code

But wait! What can you do to make sure that your web application works in legacy browsers? Also, what to do if any of your application’s dependencies use modern features of JavaScript that aren’t supported by popular browsers? 

In both cases, you need to transpile the code of the dependencies.

### Manual transpilation

Let’s assume that we’re using webpack and [babel-loader](https://webpack.js.org/loaders/babel-loader/#usage). Often, the config would look like this: 

```javascript
module: {
  rules: [
    {
      test: /\.jsx?$/,
      exclude: /node_modules/,
      use: {
        loader: 'babel-loader',
        options: {
          presets: [
            ['@babel/preset-env', { targets: 'defaults' }]
          ]
        }
      }
    }
  ]
}
```

It’s suggested in the documentation and usage examples for Babel and `babel-loader` to exclude `node_modules` from transpilation (`exclude: /node_modules/`) to optimize the performance.

By removing the `exclude` rule, we’ll **enable the transpilation of dependencies’ code** in exchange for the increased bundling time. By providing a custom function as the `exclude` rule, we also can transpile just a subset of all dependencies:

```javascript
exclude: _ => /node_modules/.test(_) && !/node_modules\/(nanostores|p-limit)/.test(_)
```

We can also select files by their filename extensions:

```javascript
exclude: _ => /node_modules/.test(_) && !/(\.babel\.js|\.mjs|\.es)$/.test(_)
```

### Manual transpilation — the benchmark

Let’s check how `babel-loader` configuration affects the bundle size and bundling time. Consider an application with three very different dependencies:

- [svelte](https://www.npmjs.com/package/svelte) (uses modern features of JavaScript such as arrow functions)
- [p-limit](http://npmjs.com/package/p-limit) (uses bleeding-edge features of JavaScript such as private class fields)
- [axios](http://npmjs.com/package/axios) (regular ES5 code)

| Config | Transpilation | Compatibility | Bundle size | Bundling time |
| --- | --- | --- | --- | --- |
| Basic | — | No way to predict which web browsers will work with this bundle | 21 KB | 1.8 s |
| `target: defaults and supports es6-module` | To ES6 code. Private class fields will be downgraded, arrow functions and classes will remain as is | Modern browsers  | 22 KB | 2.6 s |
| `target: defaults` with polyfills | To ES5 code | All browsers | 123 KB | 6.1 s |

You can see that the total bundling time for modern browsers and all browsers with `babel-loader` is 8.7 s. Please also note that the basic, non-transpiled bundle won’t work with legacy browsers because of `p-limit`.

(By the way, I also have a [blog post](https://dev.to/dangreen/speed-up-with-browserslist-30lh) that explains in detail how to build several bundles for different browsers.)

Okay, but what if you don’t want to tinker with configs and specify files and packages to be transpiled manually? Actually, there’s a readily available tool for that!

### Transpilation with optimize-plugin

Meet [optimize-plugin](https://github.com/developit/optimize-plugin) for webpack by Jason Miller from Google ([@_developit](https://twitter.com/_developit)). It will take care of everything and even more:

- It will transpile your application’s source code and the code of all dependencies.
- If needed, it will generate two bundles (for modern and legacy browsers) using the [module/nomodule pattern](https://jasonformat.com/modern-script-loading/).
- On top of that, it can also upgrade ES5 code to ES6 using [babel-preset-modernize](https://github.com/developit/babel-preset-modernize)!

Let’s see what `optimize-plugin` will do to our example application with three dependencies:

| Config | Transpilation | Compatibility | Bundle size | Bundling time |
| --- | --- | --- | --- | --- |
| Basic | To ES6 code. Also, to ES5 code with polyfills | All browsers | 20 KB for modern browsers. 92 KB for legacy browsers (including 67 KB of polyfills) | 7.6 s |

The total bundling time with `optimize-plugin` is 7.6 s. As you can see, `optimize-plugin` is not only faster than `babel-loader`, but it also produces a smaller bundle. You can check my results using the code from my [optimize-plugin-demo](https://github.com/dangreen/optimize-plugin-demo) repository.

### Why optimize-plugin wins

The performance boost is possible because the code is analyzed and bundled only once. After that, `optimize-plugin` transpiles it for modern and legacy browsers.

Smaller bundle size is possible thanks to [babel-preset-modernize](https://github.com/developit/babel-preset-modernize). Chances are that you use ES6+ features in your application’s code but you never can predict which features are used in the code of the dependencies. Since `optimize-plugin` works with the bundle that already has the code of all dependencies, it can transpile it as a whole.

Here’s how `babel-preset-modernize` works. Consider this code snippet:

```javascript
const items = [{
  id: 0,
  price: 400
}, {
  id: 1,
  price: 300
}, {
  id: 2,
  price: 550
}];
const sum = items.reduce(function (sum, item) {
  const price = item.price;
  return sum + price;
}, 0);

console.log(sum);
```

After transpilation to ES6, we’ll get this code:

```javascript
const items = [{
  id: 0,
  price: 400
}, {
  id: 1,
  price: 300
}, {
  id: 2,
  price: 550
}];
const sum = items.reduce((sum, {
  price
}) => sum + price, 0);

console.log(sum);
```

Here’s what has changed:

- A regular anonymous function was upgraded to an arrow function.
- `item.price` field access was replaced with the function argument destructuring.

Code size shrinked from 221 to 180 bytes. Note that we applied only two transformations here but `babel-preset-modernize` can do a lot more.

## What’s next?

`optimize-plugin` works really great but it still has some room for improvement. Recently, I’ve [contributed](https://github.com/developit/optimize-plugin/releases/tag/1.1.0) a few pull requests, including the support for webpack 5.

If `optimize-plugin` looks promising to you, I encourage you to [give it a try](https://github.com/developit/optimize-plugin#install) in your projects and maybe contribute some improvements as well.

Anyway, starting today, please always transpile the code of the dependencies, whether with `optimize-plugin` or not, to make sure that you have full control over your application’s compatibility with modern and legacy browsers. Good luck!

Best Practices

The Importance of Data Democratization

Building Business Intelligence Dashboards (your team will actually want to use)

Get Cube updates to your inbox

The (Earnest) Importance of Orchestrating Security Context in a Semantic Layer: HIPAA and SOC 2

How to build tree-shakeable JavaScript libraries

Why and how to transpile dependencies of your JavaScript application

Upgrade your data stack today