Ever since the wide availability of ChatGPT in early 2023, Large Language Models (LLMs) have become a subject of intense interest and investment in nearly every industry. As we all try to wrap our heads around exactly how the power of generative AI will transform the world, here at Cube it was obvious to us that semantic layers would have a part to play. This would take shape as the integration of the Cube semantic layer into applications that retrieve information from data stores, perform analysis and return requested results to a user who is interacting with them using natural language, interpreted by an LLM. The exact return method of these results could be multimodal - natural language generation, tabular result sets, or even chained into visualization libraries to return dashboards, but the big idea is “ask a question, get an answer”. We weren’t sure exactly what it would look like, but we knew it was coming.

To prepare for this tectonic shift, we focused on learning about how builders were thinking about this problem and we released features that we felt would empower them such as our semantic document loader for LangChain and our expanded meta endpoints to make it easier to provide the useful context within Cube to the LLM, improving consistency and reducing hallucinations. We also engaged with many builders focusing on next generation analytics experiences driven by natural language query interfaces enabled by LLMs such as our partners at [Patterson Consulting](https://pattersonconsultingtn.com/) and [Delphi Labs](https://www.delphihq.com/) who have been instrumental in delivering AI-enabled analytics use cases for our customers. 

While we all felt that it made sense that an LLM based application would provide more accurate results when answering questions that required examining data within a database with the aid of well designed semantic layer, when compared with more simple zero-shot text-to-sql methods, we will be the first to tell you that we were long on customer anecdotes and light on empirical evidence. 

Recently a group of researchers from data.world (Juan F. Sequeda, Dean Allemang and Bryon Jacob) have published a [ground-breaking paper](https://arxiv.org/pdf/2311.07509.pdf) studying the accuracy of LLMs answering questions with a text-to-sql pattern and comparing those results with the accuracy of introducing a knowledge graph into the process. For those who may not be familiar, knowledge graphs are somewhat similar to semantic layers in that they are also an additional layer of business context that can inform usage of data. 

In addition to the results, the paper also puts forward the detailed methodology for evaluating the accuracy of LLMs in answering business questions asked of a database. They provide a schema, sample data, natural language questions and expected answers in a format that is fairly easy to replicate their experiment. Our partners at Delphi, who build an AI-enabled analytics application on top of semantic layers, quickly realized that this methodology could be adapted to perform a benchmark of the accuracy of their own solution. **[They have recently published some interesting findings of their own](https://delphihq.substack.com/p/delphi-at-100-video), that they were able to achieve an outstanding 100% accuracy with Delphi connected to a subset of the data.world test modeled in Cube as the semantic layer.** This is an incredible result and not only a validation of the ability of semantic layers to provide necessary context to improve LLM accuracy, but also speaks to their thoughtful prompt engineering and design choices. 

## Why are semantic layers the missing piece?

The two driving themes for why we need to use semantic layers with LLMs are context and constraint. We see more and more teams moving towards providing context through prompt engineering and doing things like providing the LLM with a knowledge graph to interface with. This is definitely a step in the right direction, but you also need to constrain their output.

Semantic layers provide both a knowledge graph and a constrained interface for an LLM. As a special type of knowledge graph, a semantic layer provides the LLM with a model of a world, composed of entities and their measures and dimensions. LLMs have been trained on language, and all language is composed of entities (nouns) and their dimensions and attributes (adjectives). Language also contains mathematical terms such as ‘total’, ‘running total’, ‘average’… as a result, LLMs are well-suited to knowledge graphs. 

The reason LLMs are also quite good at SQL is because of how many documents and articles (think Stack Overflow) contain SQL. However, SQL is a minuscule fraction of what LLMs have been trained on. An interface closer to natural language is a better fit for LLMs, and a constrained one, to reduce chance of error and hallucination, is even better. Requests to Cube’s REST API are both close to natural language (a requested list of metrics, dimensions and filters with few, if any, additional terms) and highly constrained (there is only really one way to form a request), if the LLM generates the request incorrectly this will almost always result in an error rather than a hallucination.

However, this has been theory so far, with some anecdotal evidence from using Delphi. What we needed was a benchmark, and now that the data.world one has been created - we have one.

## Testing Methodology
We undertook an ambitious benchmark test, following the footsteps of data.world and dbt Labs, who [tested their semantic layer on a subset of 11 questions](https://github.com/dbt-labs/semantic-layer-llm-benchmarking/tree/main) from the original benchmark. Our goal was to replicate dbt Labs’s methodology on our platform, utilizing Delphi in conjunction with the Cube semantic layer. **The outcome was remarkable - while dbt Labs achieved an impressive 83% across 8 questions (excluding 3 with multi-hop joins), Delphi + Cube reached a perfect score of 100% on all questions.**

The benchmark began by setting up a Snowflake data warehouse. We created the necessary tables from the data.world benchmark using the [DDL available in our GitHub repository](https://github.com/Delphi-Data/delphi-semantic-layer-llm-benchmark/blob/main/ACME_small.ddl).

Once the database structure was in place, we proceeded to populate it with the data files sourced from the insurance dataset [available on data.world’s GitHub](https://github.com/datadotworld/cwd-benchmark-data/tree/main/ACME_Insurance/data), originally developed by Object Management Group. This model, which involves a 3NF structure with multi-hop joins, was crucial in replicating the complexity of real-world data, setting the stage for an authentic benchmarking experience.

Next, we started building out the Cube semantic layer. This involved signing up for Cube’s free tier and creating a new project using the credentials from our Snowflake account. Using Cube Cloud’s built-in IDE, we developed the Cube schema available in our GitHub repository, including all tables and dimensions from the benchmark (even “extra” ones our questions did not use) and some basic measures.

Finally, we created a new free account on Delphi, connected Cube via its REST API integration, and then were off to the races. Working off the benchmark results sheet created by dbt Labs, we ran each question in a separate Delphi conversation and tracked the results. We replicated this 5 separate times to ensure that our results were consistent and not impacted by the non-determinism that’s often possible with LLMs.

Our results are publicly available, including the Cube queries Delphi generated. Delphi was able to answer 100% of the questions correctly, even including some where it was able to calculate the needed measures on the fly.

In line with our commitment to transparency and community collaboration, we have made our Cube schema and benchmark instructions publicly available. These resources can be accessed on our GitHub repository at [Delphi Semantic Layer LLM Benchmark](https://github.com/Delphi-Data/delphi-semantic-layer-llm-benchmark), inviting others to replicate our benchmark and share their findings.

## Conclusion

There is now empirical evidence that using a semantic layer with an LLM, for analytics, is superior to other methods. We have chained a number of proprietary methods in addition to this to deliver the best-in-class performance we see with Delphi. We invite everyone to take a look at the testing methodology above and take both [Cube](https://cubecloud.dev/auth) and [Delphi](https://auth.delphihq.com/en/signup) for a spin. We’d love to show you how easy it is to start asking questions of your data and feel confident that the answers your team will receive are grounded in the reality of your data.


Brian Bickell

Artificial Intelligence

Semantic layers provide LLMs context and constraint.

Semantic Layers are the missing piece for AI-Enabled Analytics

While it may seem like self-hosting Cube OSS is the obvious choice (It’s free after all, right?), it's important to recognize the cost of infrastructure, resources, and maintenance that comes with this “free” choice. Here we attempt to break down those costs to help with your evaluation.

In this analysis we’ll look at three example deployment scenarios for Cube:

1. Just Testing (Small)
2. Going into Production (Medium)
3. Scaling to Enterprise (Large)

## But First, Why is Total Cost of Ownership (TCO) Important?

The Total Cost of Ownership (TCO) is a pivotal metric in any technology decision-making process for several compelling reasons. First, it’s crucial to dispel the misconception that self-hosting and internally supporting Open Source Software is entirely free. While self-hosting OSS may not require upfront licensing fees, a TCO analysis accounts for _**all**_ associated costs, including deployment, maintenance, and support, to give a more comprehensive view of what you should expect to spend.

Additionally, a TCO analysis captures long-term savings or expenses by looking at more than just the initial cost of deployment. Oftentimes, streamlined operations can translate into enhanced efficiency and reduced overhead over a longer time horizon.

Finally, most OSS products are lacking important features that can deliver productivity enhancements to the business - further enhancing the benefit of and TCO calculation.

Let’s take a closer look at the three deployment scenarios to understand how Cube Cloud’s TCO stacks up against self-hosting Cube OSS.

### Scenario 1: Just Testing

When you’re initially exploring and developing with [Cube](https://cube.dev/docs/product/getting-started), your query volume is low, your resource requirements are low and you are focused on testing functionality and building a data model that works. Your front-end team will want to test integrations with their tooling and connections from their data apps. It’s also common to utilize a few pre-aggregations at this phase to improve performance for sluggish queries.

On self-hosted [Cube OSS](https://cube.dev/docs/product/getting-started/core), you may choose to deploy a single container locally or put it on a small VM so that you can collaborate and your team can connect to it. That would typically take a team several hours or days to learn the in’s and out’s of Cube configuration, then spin up and manage the instance. 

On [Cube Cloud](https://cubecloud.dev/auth), you can sign-in to the free tier in seconds which most likely has what you need for an evaluation.  Not only does this save you 100% of hosting, setup, and maintenance costs, but **you can also get started much faster.**  When you are ready to go into production, it’s just a click to get a production cluster spun up on the Starter plan. And, even on the Starter plan paying monthly, the TCO would still be less - about half as much as Cube OSS.

*Note from the Cube team: If you want to evaluate additional functionality from our higher tier plans, just [contact sales](https://cube.dev/contact) and we’ll set you up with a limited time trial.* 

![](https://ucarecdn.com/16b63ab7-86c0-412f-b061-c6d71960818e/)

### Scenario 2: Going to Production

Launching your first [Cube deployment](https://cube.dev/docs/product/deployment/production-checklist) is an exciting and crucial milestone that must be executed correctly to establish trust with your users.  You’ll need a stable Cube deployment with high-availability/redundancy for the API and Cube Store worker nodes. You’ll also need a process and tooling for testing and pushing changes to production without taking the deployment offline (CI/CD and blue/green). A [Cube Store](https://cube.dev/docs/product/deployment/production-checklist#set-up-cube-store) cluster is necessary to run pre-aggregations and it needs to be sized large enough to execute queries against your largest rollups. The hosting cost is higher than a Cube cluster that only does query pass-through.

On self-hosted Cube OSS, you may set up a Kubernetes cluster with 2 deployments - one for Blue and one for Green. Your testing and development work happens on whichever deployment is not serving the production workload. Your CI/CD orchestration pipeline would need to deploy new commits to the non-prod deployment, build or refresh any pre-aggregations in Cube Store that need to be updated, then cutover the traffic routing to the newly updated cluster to complete the code deployment.  You likely also need small dev deployments to work collaboratively on feature and dev branches, which requires additional lightweight Cube instances.

On Cube Cloud, you have built-in redundancy and there’s no Kubernetes to setup or manage. CI/CD orchestration runs transparently while you simply merge/commit changes to the main branch. You also have access to chat and email support to help you move faster and more confidently.

![](https://ucarecdn.com/72a791cc-fa34-4e31-9bb5-e08076189d2f/)
###### *Link to [production checklist](https://cube.dev/docs/product/deployment/production-checklist#appropriate-cluster-sizing) referenced above.*

### Scenario 3: Production at Enterprise Scale 

Things are going well and Semantic Layer adoption is taking off! Cube has become mission-critical to your data teams and products. Data residency requirements also dictate the need for two independent regional deployments. Fortunately, Cube can scale to meet high-traffic demands, insulating your data warehouse from costly overruns and providing consistent data models, access control, cache, and [APIs](https://cube.dev/product/data-apis) from multiple data sources to many data consumer tools. 

With this increased demand comes increased scrutiny.  Quick resolution of any data pipeline issues is critical.  Infrastructure costs must be both predictable and minimized, but performance and availability must remain high. The stakes are high, and Cube is up to the challenge.

On self-hosted Cube OSS, while the baseline functionality scales up well, additional tooling is required to best serve the mission-critical use case. These features would need to be developed and managed alongside a self-hosted Cube OSS deployment:
- CI/CD orchestration pipeline
- Application Performance Monitoring stack 
- Cluster auto-scaling and auto-recovery
- Responsive, Cube-knowledgeable DevOps support

On Cube Cloud, your deployments are ready to go on day one with all the scaling up functionality built into Cube Cloud already. Clusters can auto-scale and auto-suspend with just a click, and the billing dashboard provides immediate clarity on consumption tracking. Request monitoring is built in so you can see the performance of your API responses as they stream in, and logs can be directed to your sink of choice. Cube Cloud comes with a performant custom-built APM tailored to Cube development needs that's simply not available in OSS. This APM implementation helps Cube Cloud customers dramatically improve their development velocity. You have the full support of Cube’s infrastructure management team and a dedicated Customer Success Manager to keep things running smoothly.  

 ![](https://ucarecdn.com/1d9634a9-1258-4ada-965f-ca56cbf86590/)
 ###### - Self-hosting costs are estimated using [EC2 Linux on-demand rates](https://aws.amazon.com/ec2/pricing/on-demand/) for Scenario 1 and 2 (non-autoscaling).  Scenario 3 uses [AWS Fargate on-demand pricing](https://aws.amazon.com/fargate/pricing/) for the average cluster size.
###### - Hourly services rates ($150/hr for small, $200/hr for medium/large) taken from an informal poll of Devops Consultants.

### Comparing Apples to Apple Pie

Below is a brief comparison of how we turn the raw ingredient of Cube OSS (like apples) into the feature rich, production-ready Cube Cloud (akin to a delectable apple pie).

![](https://ucarecdn.com/27d8948a-7779-4f76-8517-a592294d3d11/)

### What If Your Data Must Stay on Your Network?

If your contractual obligation or corporate preference is to retain your data in your own infrastructure, then we have you covered with “Bring Your Own Cloud” (BYOC). Before jumping straight to this option, however, most customers in this scenario choose to list Cube as a subprocessor and can still comply with data residency requirements (e.g. European data can’t leave Europe) with dedicated regional deployments and VPC Peering.

BYOC is simply Cube Cloud deployed in a public cloud account you own and control. The Cube team still manages the infrastructure and supports the deployment, and you get the full benefit of Cube Cloud without any data being stored off premises.

#### Summary: Regardless of stage or size of deployment, running your Semantic Layer on Cube Cloud costs less than Self-Hosting Cube OSS

![](https://ucarecdn.com/7592522a-3c58-46d4-bb7a-eaf435a5c473/)

For dev and test workloads, small production use cases, and enterprise scale deployments, Cube Cloud **consistently saves you both time and money.** Cube Cloud is supported, fully-managed Cube plus key features to accelerate your development workflow while providing a predictable cost structure. Instead of building out your own self-hosted deployment, Cube Cloud is ready to power your production workload immediately. 

We fully believe in the power of open source software and are proud to provide Cube as an open source product. We’ve also created Cube Cloud to provide an even better value to an organization looking to increase speed-to-value, reduce cost overall, and reduce overhead maintaining bespoke integrations. While these are just three common deployment scenarios, there are many other configurations that could be a better fit for your organization. We’d be happy to chat with you about your use case and show you how Cube Cloud is so much more capable and even more affordable than self-hosting Cube OSS.


Tony Kau

All Things Cube

Evaluating the cost of Cube Cloud to self-hosting Cube OSS

Building your data stack: Cube Cloud or OSS 


The promise of a semantic layer is to bring consistency to interactions between data sources and data tools. The core functionality of a semantic layer is to make it easy to build data models - defining dimensions, measures, and metrics - that deliver consistent metrics to all of an organization’s data experiences. A fully-featured semantic layer includes security and data access controls as well as pre-aggregations and query caching which guarantees that consumers of data have a secure, consistent, and super fast experience. Metrics and business logic should be the same from tool to tool. Access control and performance characteristics should be the same as well. 

To deliver consistently on the promise of semantic layers, solution providers must support the widest range of data sources and data applications. A semantic layer may have an amazing set of functionality, but if it doesn’t integrate with the rest of the data stack it is unlikely to provide much value. Unfortunately for the developers of semantic layers, there is an ever-expanding set of technologies that customers expect to integrate with. One of my colleagues recently remarked “No one said it was going to be easy” and while I agree with him, **there is something we can adopt from other areas of technology with competing implementations: standardization.**

Development of an open standard for semantic layers could provide needed consistency on the subject of how business intelligence tools, data science notebooks, embedded analytics systems, catalogs and other consumption methods should interact with the semantic layer. New data applications are in development right now that would benefit from more defined integration patterns to inform their design if they want to treat the semantic layer as a first class source. Standards reduce the burden of software vendors to add semantic layer support and also benefit consumers by reducing switching cost and vendor lock-in. For this reason, we also believe that is important that the standard be an *open* standard that is not beholden to any one vendor.

**What should a standard cover?**
We believe that there are three major areas that an open semantic layer standard should cover.

- Specification of Objects
- Querying Protocols
- Metadata Exchange Protocols

### Specification of Objects: Metrics-centric, Dataset-centric

While there are many differences in internal semantic layer object representations - code-based vs UI-based, supported data modeling languages and specific object definition syntax, we are most interested in the outward representation of the semantic layer. With this in mind, there are currently two main approaches on the market for designing semantic layers: metrics-centric and datasets-centric.

**Metrics-centric.** In the metrics-centric approach, metrics are first-class objects. The benefit is that it’s closer to how people think and talk about data: we look at metrics, analyze them, and set them as KPIs. The biggest challenge with this approach is that most data consumption tools, including BIs, don’t have a notion of metrics. These BI tools operate on tables or cubes with measures and dimensions. 

![](https://ucarecdn.com/3eedf499-11f1-4ad2-a0fc-36ed10795961/)

**Dataset-centric.** The dataset-centric semantic layer exposes tables containing measures (metrics) and dimensions as first-class objects. While it might look like a more complicated framework, it provides the benefit of better flexibility and compatibility with the existing suite of tools.

![](https://ucarecdn.com/6441276a-104f-4a1d-a120-028ef2e78b2d/)

At Cube, we believe in a **dataset-centric approach**. It enables a wider interoperability with the existing data ecosystem, because every data tool is tabular data centric. It also gives flexibility to design outward representation of the semantic layer either as **entity-first** or as **metrics-first**, effectively **giving data teams the best of the two worlds.**

In the **Entity-first** approach, entities are denormalized tables, containing many measures and dimensions needed to fully describe the entity. In Cube, we use [views](https://cube.dev/docs/reference/data-model/view) to build outward representation of the semantic layer. The example below illustrates view design to support entity-first approach.

```yaml
views:
  - name: orders_view

    cubes:
      - join_path: orders
        includes:
          - status
          - created_at

          - completed_count
          - count
          - total_amount
          - average_order_value

      - join_path: orders.users
        prefix: true
        includes: 
          - city
          - age
          - gender
```

In the **Entity-first** approach, metrics tables are denormalized tables, containing one measure (metric) and all the relevant dimensions. In Cube, that can be achieved by creating a view for the specific metric, like `average_order_value` in the example below.

```yaml
views:
  - name: average_order_value

    cubes:
      - join_path: orders
        includes:
          - average_order_value
          - status
          - created_at

      - join_path: orders.users
        prefix: true
        includes: 
          - city
          - age
          - gender
```
You can learn more about [designing metrics with Cube in the documentation](https://cube.dev/docs/product/data-modeling/overview). 

### Querying Protocols: SQL and GraphQL

Semantic layer should provide a full-fledged querying interface to fulfill the promise of delivering metrics to all data consumption tools. Currently, most of the data tools use SQL protocol to query data from databases and data warehouses. Considering that, the semantic layer solution should expose a SQL endpoint so all existing tools can connect to it and query metrics. Additionally, it is instrumental that the semantic layer provides an HTTP-based interface for customer apps development and embedded analytics. [GraphQL](https://graphql.org/) is gaining popularity recently as a open-source data query and manipulation language for APIs, so it makes sense for the semantic layer to support it as an HTTP-based API.

![](https://ucarecdn.com/b3f21beb-ee1b-4249-944a-eac8bba5a5dc/)

The challenge to using SQL as a querying interface for the semantic layer is the lack of a notion of metric in the SQL language. SQL is designed to work with tabular, not multidimensional data.
That would require to extend SQL with a MEASURE type and corresponding aggregate function to query measures.

```sql
SELECT
  status,
  MEASURE(average_order_value)
FROM orders_view
GROUP BY 1
```

Supporting querying metrics with SQL will enable interoperability with a wide range of BIs and visualization tools that are already using SQL to query data from data warehouses.

### Metadata Exchange Between the Semantic Layer and the Front End

While there are many benefits to having the semantic layer decoupled from the various BI tools and data apps, it does bring on challenges as well. One of the biggest challenges of implementing a standalone semantic layer is the connection with the front-end UI and user experience.  When the semantic layer is integrated as a component of a specific BI tool, it influences and seamlessly works with the UI and end-user experience. For example, when you define an Explore object with LookML (Looker’s integrated semantic layer) it appears in the UI for end users so they can navigate and explore them. It is a challenge to provide the same level of deep integration for the decoupled semantic layer as every BI tools front-end user interface has different names, components, and functionality. 

![](https://ucarecdn.com/bdb4ecc6-f03c-444e-af10-f09800093f8b/)

For the industry to truly reach the goal of delivering consistent and accurate data into all of an organization’s BI tools and data apps, then this integration needs to be improved. If Semantic layer and BI vendors work together to build a more standard connection, this will allow better and deeper integration between the front-end and the semantic layer – thus giving the end user the best possible user experience across all BI tools at a company. 

### Join the Conversation

At Cube, we believe semantic layers need exceptional interoperability to give users choice. We endeavor to support the widest ecosystem of data sources and data consumers that we can for our customers. But the best way to do that is to work more broadly with the data community.

To that end, we are beginning discussions on the design of an open semantic layer standard and are eager to include other semantic layer vendors, business intelligence and dashboard tool vendors and any other data product tools vendors in the conversation. By doing this work collectively and transparently, we can build a better future for our users and the data community.

Artyom Keydunov

Data Engineering

The Case for a Standard

The Need for an Open Standard for the Semantic Layer 

Today, we're happy to support Looker users switching over to Cube and present `lkml2cube`, a tool for converting [LookML](https://cloud.google.com/looker/docs/reference/lookml-quick-reference) models into the Cube data model. With this tool, you can easily migrate your existing [Looker](https://cloud.google.com/looker/) projects to Cube.

## Benefits of using Cube

Cube is a code- and API-first semantic layer that enables data engineers and developers to make their data consistent, secure, performant, and accessible across every application. Cube is commonly used to implement a *semantic layer*: a middleware between your data source and your application that provides a unified view of your data, regardless of where it is stored or how it is formatted.

By migrating your LookML models to Cube, you can achieve the following:

- **Reduced development time.** Once you have migrated your LookML models to Cube, you can start using Cube's powerful features, such as its pre-built calculations, dimensions, and measures, to create your data applications. Using Cube can save you a significant amount of development time.
- **Improved data quality.** Cube's semantic layer can help you improve your data quality by performing data transformations, validations, and aggregations. Using Cube's version-controlled semantic layer definition can help you to ensure that your data is consistent, accurate, and reliable.
- **Enhanced performance.** Cube's in-memory data store can help improve your data applications' performance by caching frequently accessed data, thus providing faster user response times.
- **Increased accessibility.** Cube's API-first design easily integrates Cube into your existing data applications.

Initially, we’ve created `lkml2cube` to facilitate our work while supporting the migration of Looker users to [Cube Cloud](https://cube.dev/product/why-cube-cloud). Now, we’re happy to put it into [open source](https://github.com/cube-js/lkml2cube) to assist more Looker users in their migration over to Cube.

Given Cube’s current position as a [leader and fast-mover](https://cube.dev/blog/cube-recognized-as-a-leader-and-fast-mover-by-gigaom) among semantic layers, there’s no surprise we see more and more companies migrating to Cube from their legacy solutions:

[![](https://ucarecdn.com/d266a78b-3dcf-4355-a59d-17a15007afa1/)](https://cube.dev/gigaom-rates-cube-as-a-leader-for-semantic-layers)

Here’s how one of Cube Cloud users describe their experience:

> We recently transitioned from Looker to Cube in an effort to enhance the user experience on our data-intensive internal platform, both in terms of performance and user interface. Our aim was to seamlessly embed data and analytics into our team's workflows.
> 
> With the invaluable assistance and robust tools provided by the Cube team, we successfully migrated a significant portion of our LookML project, encompassing hundreds of views and explores, to Cube’s data modeling language in just a matter of weeks.
> 
> *Director, Data Products at an investment firm*

## Migration example

Cube’s data modeling language has a few similarities with LookML, however, it’s arguably more powerful, flexible, and fit as a data modeling language for a modern semantic layer.

* [Cubes](https://cube.dev/docs/reference/data-model/cube) represent tables of data in Cube and are roughly equivalent to [views](https://cloud.google.com/looker/docs/reference/param-view) in LookML. However, cubes are a bit more powerful since they can contain [join](https://cube.dev/docs/reference/data-model/joins) and [pre-aggregation](https://cube.dev/docs/reference/data-model/pre-aggregations) definitions—the latter are used to provide granular query acceleration configuration and inform Cube’s [aggregate awareness](https://cloud.google.com/looker/docs/aggregate_awareness).
* [Views](https://cube.dev/docs/reference/data-model/view) sit on top of the data graph of cubes and create a facade of your whole data model; they are roughly equivalent to [explores](https://cloud.google.com/looker/docs/reference/param-explore) in LookML. However, when referencing cubes, views allow to define an exact full join path and remove the ambiguity of searching the join path within the data graph. `lkml2cube` accounts for this quirk of LookML and performs a breadth-first search in the data graph to find the most suitable join relationship.

Consider a simple Looker project that is structured in the following way: 

```
.
├── my_explores.model.lkml
└── views
    ├── line_items.view.lkml
    ├── orders.view.lkml
    ├── product_categories.view.lkml
    ├── products.view.lkml
    ├── suppliers.view.lkml
    └── users.view.lkml
```

After applying `lkml2cube`, an equivalent Cube project would be structured similarly: 

```
.
└── model
    └── cubes
        ├── line_items.yml
        ├── orders.yml
        ├── product_categories.yml
        ├── products.yml
        ├── suppliers.yml
        └── users.yml
    └── views
        └── orders_view.yml
```

You can generate the Cube data model for your entire Looker project by executing a single command against the file with LookML explores. `lkml2cube` would inspect that file, follow the references in the `include`parameter, and generate necessary cubes and views:

```bash
lkml2cube views my_explores.model.lkml --outputdir model/
```

For example, the following `views/orders.view.lkml` file…

```javascript
view: orders {
  view_label: "Orders"

  sql_table_name: "public.orders" ;;

  dimension: unique_key {
    type: string
    primary_key: yes
    sql: ${TABLE}."id" ;;
  }

  dimension: status {
    type: string
    sql: ${TABLE}."status" ;;
  }

  dimension_group: created_at {
    type: time
    timeframes: [
      raw,
      time,
      date,
      week,
      month,
      quarter,
      year
    ]
    sql: ${TABLE}."created_at" ;;
  }

  measure: count {
    type: count ;;
  }

  measure: total_order_amount {
    type: sum
    sql: ${TABLE}."number" ;;
  }
```

...would be converted into the following `model/cubes/orders.yml` file:

```yaml
cubes:
  - name: orders
    sql_table: public.orders

    joins:
      - name: users
        sql: "{CUBE}.user_id = {users}.id"
        relationship: many_to_one

      - name: products
        sql: "{CUBE}.product_id = {products}.id"
        relationship: many_to_one

    dimensions:
      - name: id
        sql: "{CUBE}.id"
        type: number
        primary_key: true

      - name: status
        sql: "{CUBE}.status"
        type: string

      - name: created_at
        sql: "{CUBE}.created_at"
        type: time

    measures:
      - name: count
        type: count

      - name: total_order_amount
        sql: "{CUBE}.number"
        type: sum
```

Consequently, the following `my_explores.model.lkml` file...

```javascript
explore: orders {
  label: "Orders Summary"

  join: users {
    relationship: many_to_one
    sql_on: ${orders}.user_id = ${users}.id ;;
    type:  left_outer
  }

  join: products {
    relationship: many_to_one
    sql_on: ${orders}.product_id = ${products}.id ;;
    type:  left_outer
  }

  join: line_items {
    relationship: one_to_many
    sql_on: ${orders}.id = ${line_items}.order_id ;;
    type: left_outer
  }

  join: product_categories {
    relationship: many_to_one
    sql_on: ${products}.product_category_id = ${product_categories}.id ;;
    type: left_outer
  }

  join: suppliers {
    relationship: many_to_one
    sql_on: ${products}.supplier_id =${suppliers}.id ;;
    type: left_outer
  }
}

```

...would be converted into the following `model/views/orders_view.yml` file:

```yaml
views:
  - name: orders_view

    cubes:
      - join_path: orders
        includes: "*"

      - join_path: orders.users
        prefix: true
        includes: "*"

      - join_path: orders.products
        prefix: true
        includes: "*"

      - join_path: orders.line_items
        prefix: true
        includes: "*"

      - join_path: orders.products.product_categories
        prefix: true
        includes: "*"

      - join_path: orders.products.suppliers
        prefix: true
        includes: "*"
```

Note unambiguous join paths like `orders.products.product_categories` and `orders.products.suppliers` in the example above that `lkml2cube` was able to detect and express in the Cube data model. Over time, it would save hours, if not days, of work when building and debugging complex data models.

## Conclusion

`lkml2cube` is a powerful tool that can help you convert LookML models into Cube’s data modeling language and assist your migration from Looker to Cube. In case this tool is relevant to your data stack, please feel free to [check the documentation](https://cube.dev/docs/reference/python/lkml2cube) and give it a try.

Lastly, you're always very welcome to join our [Slack community](http://slack.cube.dev/) of more that 9,000 data practitioners and share your feedback on this update.

Paco Valdez

Looker

Supporting the transition to Cube's semantic layer from legacy solutions

Introducing a tool for Looker to Cube migration

Risen from the ashes of the [Great Unbundling of Airflow](https://blog.fal.ai/the-unbundling-of-airflow-2/), data transformation tools serve an important role in modern data pipelines. (Note that I don’t dare to say “[modern data stack](https://davidsj.substack.com/p/the-modern-data-stack-is-dead-long-61f)” in late 2023.)

Among [very promising](https://sqlmesh.com) peers, [dbt](https://www.getdbt.com/) is still the most popular data transformation tool, with almost [8,000 stars on GitHub](https://github.com/dbt-labs/dbt-core), a vibrant community, and a lot of impact on the current state of data engineering; more on that below.

## Data transformation upstream of the semantic layer

Nowadays, no one would be surprised to see a data transformation tool being used to define hundreds or thousands of [staging](https://docs.getdbt.com/guides/best-practices/how-we-structure/2-staging) and intermediate models, mince the data through a complex DAG, and finally output it to a warehouse as [data marts](https://docs.getdbt.com/guides/best-practices/how-we-structure/5-semantic-layer-marts).

Similarly, virtually no one would disagree that direct access to transformed data in the warehouse is the root of all evil. It leads to duplicated effort, disparate calculated metrics, poor performance, incoherent access controls, and generally unhappy data teams and business users. Luckily, these are exactly the issues that the semantic layer is designed to solve.

![](https://ucarecdn.com/d034e243-108b-4f93-9e40-f8da224f75c4/)

With a semantic layer, you would define your domain-specific data model on top of data marts, use it as the single source of truth for metric definitions, leverage consistent access controls and aggregate awareness, and expose metrics to all kinds of data consumers ([BI tools](https://cube.dev/use-cases/connected-bi), [AI agents](https://cube.dev/use-cases/llm-and-ai-semantic-layer), and [data apps](https://cube.dev/use-cases/embedded-analytics)) via a rich set of APIs:

![](https://ucarecdn.com/4f7c0d0f-ecbb-4666-bab1-54091fd04b8c/)

Here's how a few Cube users describe their approach and experience with using dbt and Cube together:

> We create a [star schema](https://en.wikipedia.org/wiki/Star_schema) with dbt and feed that to Cube. In Cube, we define metrics on top of that star schema.

> We’ve chosen to expose views managed in dbt to Cube. These views are downstreams from our fact and dimension tables. To define the views, we usually do `SELECT *` and then add the dimensions that we want to calculate on the fly.  That way, all dimension logic is built in dbt, and we use Cube to handle metric definitions and joins.

Indeed, it's very heart-warming to know that members of the Cube community are successful building their data pipelines with both tools. As a [leader and fast-mover](https://cube.dev/blog/cube-recognized-as-a-leader-and-fast-mover-by-gigaom) among semantic layers, with more than [16,000 stars on GitHub](https://github.com/cube-js/cube), Cube has proven itself as a mature and feature-rich solution that integrates and works great with all kinds of data tools, including dbt.

[![](https://ucarecdn.com/d266a78b-3dcf-4355-a59d-17a15007afa1/)](https://cube.dev/gigaom-rates-cube-as-a-leader-for-semantic-layers)

## All-new dbt & Cube integration

We’ve always wanted dbt and Cube users to have the best experience using both tools together. Recently, we have [introduced the support for Python and Jinja](https://cube.dev/blog/introducing-python-and-jinja-for-data-modeling) to Cube that laid the groundwork for the integration.

**Today, we’re excited to announce the all-new dbt integration that is based on Python and Jinja.** It comes as the [`cube_dbt` package](https://cube.dev/docs/reference/python/cube_dbt) that simplifies defining the data model in the semantic layer on top of dbt models. We are also releasing a guide on [using Cube with dbt](https://cube.dev/docs/guides/dbt) that explains the integration steps in detail.

In a nutshell, the integration involves loading the metadata of a dbt project from `manifest.json`, selecting necessarty dbt models, and exposing them to Jinja:

```python
from cube import TemplateContext
from cube_dbt import Dbt

manifest_url = 'https://cube-dbt-integration.s3.amazonaws.com/manifest.json'
dbt = Dbt.from_url(manifest_url).filter(paths=['models/marts/'])

template = TemplateContext()

@template.function('dbt_model')
def dbt_model(name):
  return dbt.model(name)
```

The only next step is to render dbt models as [cubes](https://cube.dev/docs/reference/data-model/cube) with [dimensions](https://cube.dev/docs/reference/data-model/dimensions) in YAML (with column types and primary keys automagically inferred from the dbt project metadata) and enrich the data model with [measures](https://cube.dev/docs/reference/data-model/measures), [joins](https://cube.dev/docs/reference/data-model/joins), and [pre-aggregations](https://cube.dev/docs/reference/data-model/pre-aggregations):

```yaml
{% set model = dbt_model('orders') %}

cubes:
  - {{ model.as_cube() }}

    dimensions:
      {{ model.as_dimensions() }}

    # Model-specific measures
    measures:
      - name: count
        type: count

      - name: statuses
        sql: "STRING_AGG({status})"
        type: string

    # Model-specific joins
    joins:
      - name: users
        sql: "{CUBE.user_id} = {users.id}"
        relationship: many_to_one

    # Model-specific pre-aggregations
    pre_aggregations:
      - name: count_by_users
        measures:
          - CUBE.count
        dimensions:
          - users.full_name
```

Needless to say, we encourage you to check the [guide on using Cube with dbt](https://cube.dev/docs/guides/dbt) and [see how everything works in Cube Cloud](https://cubecloud.dev/auth/signup) or Cube Core today. Support for Python and Jinja comes with the Cube Core v0.34 release (please note that macOS on M1/M2 as well as Windows are not supported yet but we’re actively working on bringing these platforms very soon).

You’re also very welcome to [join our Slack community](https://slack.cube.dev) to share your thoughts and feedback on this integration. **Finally, we’ve planned a [webinar on October 11, 2023](https://cube.dev/events/trust-your-insights-with-cube-and-dbt) to talk in-depth about the dbt integration—please don’t hesitate to come, see everything in action, and ask your questions.**

Igor Lukanin

News

Introducing dbt integration with Cube

Yesterday, we announced that [Python can be used for configuration](https://cube.dev/blog/introducing-python-for-configuration) in Cube. That’s not all: actually, Python can also be used for data modeling! So, now you can have Python-only experience with Cube, either providing programmatic configuration or authoring dynamic data models with YAML and Jinja. 

**Data modeling is arguably the most important part of Cube as a semantic layer.** The data model encapsulates everything from metric definitions and the public-facing facade of the semantic layer to the access control and aggregation-awareness configuration. Similar to models in your data transformation tools, Cube deployments often have dozens or hundreds of cubes and views in their data models.

## YAML syntax for data modeling

Almost a year ago, we [introduced the support of the YAML syntax](https://cube.dev/blog/introducing-cube-support-for-yaml-data-modeling) for data modeling. It’s a great alternative to the JavaScript-based syntax because YAML is just easier to write and read. We quickly saw the rising adoption of YAML in the Cube community.

```yaml
cubes:
  - name: revenue
    sql_table: public.revenue
    public: "{{ COMPILE_CONTEXT.security_context.is_finance }}"

    dimensions:
      - name: fiscal_year
        sql: "{fiscal_periods.year_label}"
        type: string

    measures:
      - name: ltm_revenue
        description: Last twelve months (LTM) revenue
        sql: amount
        type: sum
        rolling_window:
          trailing: 12 month

    pre_aggregations: []
```

However, pure YAML lacks the dynamicity of JavaScript that allows to define complex and powerful [programmatic data models](https://cube.dev/docs/product/data-modeling/dynamic) which depend on external data (e.g., retrieved from an API endpoint) or just remove code duplication by relying on a library of helper functions.

With the current update, we put YAML data models on par with JavaScript ones and empower you to organize the data model code, add dynamicity to it, and use programming techniques of your preference. 

## Adding Jinja and Python to the mix

**We’re glad to announce that now Cube supports [Jinja as a template engine](https://jinja.palletsprojects.com/en/3.1.x/) and Python as a programming language to define dynamic data models in YAML**

Jinja is an expressive template engine that is used by many prominent Python projects, including Flask and Python—not to mention that many data engineers are used to writing [Jinja in dbt projects](https://docs.getdbt.com/docs/build/jinja-macros). With Jinja, you can add variables, loops, macros, and other logic directly in YAML.

```yaml
{%- macro dimension(column_name, type='string', primary_key=False) -%}
      - name: {{ column_name }}
        sql: {{ column_name }}
        type: {{ type }}
        {% if primary_key -%}
        primary_key: true
        {% endif -%} 
{% endmacro -%}

{%- set metrics = {
  "mau": 30,
  "wau": 7,
  "day": 1
} %}

cubes:
  - name: hello_jinja
    sql_table: public.orders
    data_source: {{ 'bigquery' if env_var('ENV') == 'prod' else 'postgres' }}

    dimensions:
      {{ dimension('id', 'number', primary_key=True) }}
      {{ dimension('status') }}
      {{ dimension('created_at', 'time') }}

    measures:
      {%- for name, days in metrics | items %}
      - name: {{ name }}
        type: count_distinct
        sql: user_id
        rolling_window:
          trailing: {{ days }} day
          offset: start
      {% endfor %}
```

Even more importantly, Python functions can be called from Jinja templates, giving you an option to write some, or most, of your logic in Python and then render the result in YAML as cubes, views, or their parts. In the following example, the `model/globals.py` file is used to register a Python function to be callable from Jinja.

```python
from cube import TemplateContext
from cube_dbt import Dbt

manifest_url = 'https://cube-dbt-integration.s3.amazonaws.com/manifest.json'
dbt = Dbt.from_url(manifest_url).filter(paths=['models/marts/'])

template = TemplateContext()

@template.function('dbt_model')
def dbt_model(name):
  return dbt.model(name)
```

Needless to say, you can also use dependencies and benefit from the whole Python ecosystem. In the example above, we employ the `cube_dbt` package that… um, as you can guess, does something peculiar—more on that in the next blog post!

We are committed to support both Python and JavaScript for data modeling going forward. We recommend using YAML by default and adding Jinja and Python for dynamic data models as needed.

<video width="100%" controls><source src="https://ucarecdn.com/e7fb3a6d-9705-4a1b-ab87-c2af9ee93815/jinjapythondemo.mp4" type="video/mp4"/></video>

We’ve updated the [documentation on dynamic data models](https://cube.dev/docs/product/data-modeling/dynamic/jinja) with Jinja and Python code examples and explained how the `model/globals.py` file works. Please feel free to check out and try it yourself!

**Support for Python comes with the Cube Core v0.34 release.** You can [get started today in Cube Cloud](https://cubecloud.dev/auth/signup); please make sure to switch to the latest version channel. You can also get started with Cube Core; however, please note that macOS on M1/M2 as well as Windows are not supported yet; we’re actively working on introducing the support for these platforms very soon.

Please give Python a try out and [join our Slack community](https://slack.cube.dev) to share your thoughts and feedback. Also, stay tuned for further updates, including the one related to the `cube_dbt` package. **We’ve already scheduled a [webinar on October 11, 2023](https://cube.dev/events/trust-your-insights-with-cube-and-dbt) to cover that—please don’t hesitate to RSVP and come say hi!**


Introducing Python and Jinja for data modeling

TL;DR: Python has come to Cube; and this is huge. However, please bear with us for a complete story. 

We’re building Cube as a feature-rich, universal semantic layer that works great for many use cases. Consequently, Cube needs and provides a great deal of flexibility in [configuration](https://cube.dev/docs/product/configuration). On the technical side, that includes plenty of [environment variables](https://cube.dev/docs/reference/configuration/environment-variables)—quite unsurprising for a modern cloud-native application. However, some of the most powerful configuration options, usually related to multitenancy and access control, are provided programmatically in a configuration file.

Usually, real-world Cube deployment will employ both environment variables and a configuration file with dozens, if not hundreds, lines of code. Cube would use programmatic configuration to fine-tune multitenancy, [inspect and transform](https://cube.dev/docs/reference/configuration/config#query_rewrite) queries on-the-fly, or implement custom authentication to the [rich set of APIs](https://cube.dev/docs/product/apis-integrations) that Cube provides; this list can go on and on.

This is very much in sync with the [code-first principle](https://cube.dev/docs/product/introduction#code-first) that we embrace here at Cube. We firmly believe that application of software engineering best practices—like the use of version control systems, code reviews, and sound code organization—to the data engineering field moves it further in the right direction and helps create high-quality data products. Programmatic configuration is crucial to this approach.

## The Good, the Bad, and the Ugly

From its early days, Cube has supported the JavaScript programming language in the `cube.js` file for defining configuration options. It was handy, since [embedded analytics](https://cube.dev/use-cases/embedded-analytics) were the very first use case that Cube mastered. Often, it was implemented by software engineers who were quite familiar with JavaScript.

Since then, Cube has grown its audience to an exciting blend of software developers and data engineers, marked by more than [16,000 stars](https://github.com/cube-js/cube) on GitHub. Cube has also expanded supported use cases with [connected BI](https://cube.dev/use-cases/connected-bi) and, of course, [AI](https://cube.dev/use-cases/llm-and-ai-semantic-layer). With all that, we felt the urge to take Cube further to better serve its expansive, diverse, and [opinionated](https://twitter.com/pdrmnvd/status/1533876877806346240) audience.

## Introducing Python

**Today, we’re happy to announce that now Cube supports Python programming language in the `cube.py` file to define configuration options!**

Look how beautiful it is:

```python
from cube import config, file_repository

config.base_path = '/my-api'

config.scheduled_refresh_time_zones = [
  'America/New_York',
  'America/Los_Angeles'
]

@config('context_to_app_id')
def context_to_app_id(ctx: dict) -> str:
  tenant_id = ctx['securityContext'].setdefault('tenant_id', 0)
  return f"CUBE_APP_{tenant_id}"

@config('repository_factory')
def repository_factory(ctx: dict) -> list[dict]:
  tenant_id = ctx['securityContext'].setdefault('tenant_id', 0)
  return file_repository(f"model/{tenant_id}")

@config('check_auth')
def check_auth(ctx: dict, token: str) -> None:
  if token != 'my_precious_secret':
    raise Exception('Access denied, try again later')

@config('query_rewrite')
def query_rewrite(query: dict, ctx: dict) -> dict:
  context = ctx['securityContext']
  if 'user_id' in context:
    query['filters'].append({
      'member': 'users.id',
      'operator': 'equals',
      'values': [context['user_id']]
    })
  return query
```

We are committed to supporting both Python and JavaScript going forward. Please consider using Python for new Cube projects to benefit from its concise syntax, better readability, and great ecosystem.

<video width="100%" controls><source src="https://ucarecdn.com/389ced64-91e6-4387-aa90-d5cf5ca6e03b/demopy.mp4" type="video/mp4"/></video>

We’ve updated the [configuration options reference](https://cube.dev/docs/reference/configuration/config) with Python code examples and [explained](https://cube.dev/docs/product/configuration#cubepy-and-cubejs-files) how the `cube.py` file works. Please feel free to try it out yourself!

**Support for Python comes with Cube Core v0.34 release.** You can [get started today](https://cubecloud.dev/auth/signup) in Cube Cloud; please make sure to switch to the latest version channel. You can also get started with Cube Core; however, please note that macOS on M1/M2 as well as Windows are not supported yet; we’re actively working on introducing the support for these platforms in the nearest future.

Please give Python a try out and join our Slack community to share your thoughts feedback. Also, stay tuned for further updates—we have so much more to share!


Data engineering lingua franca inevitably taking over Cube

Introducing Python for configuration

Working with Databricks and Cube is a great way to ensure your data engineering team can build and maintain a unified semantic layer for your organization. At a higher level, it is a better means to make cloud data accessible and consistent to every data consumer in and outside of your company.

By combining Databricks’ unified analytics and data store with Cube’s data modeling, access control, caching, and APIs, it’s now possible to deliver consistent metrics securely to business intelligence tools, embedded analytics use cases, and AI agents.

## Why a Semantic Layer?

A semantic layer functions as a middleware component between all of your data sources, including cloud-based data lakehouses, such as Databricks, and downstream applications. Its primary role is to abstract complicated physical data models and present them in a more user-friendly manner with improved field labeling, field descriptions, and dynamically calculated metrics. It serves as a contextual bridge between the data sources and the diverse analytical tools businesses employ for data manipulation, slicing, dicing, and analysis.

![](https://ucarecdn.com/5bcadadb-60ee-4698-bc6e-1f1e2ad8aa71/)

## Why Cube and Databricks?

**Unify your data stack**. Databricks established itself as a high-performance Lakehouse for a **modern data stack**, unifying all data types. Combined with Cube, you will have consistently modeled metrics and performant queries for all your downstream data consumers. 

**Modern data stack** follows several foundational principles and architecture designs: 
* Using a Lakehouse like Databricks as a central data platform
* ELT vs. [ETL](https://www.databricks.com/glossary/extract-transform-load)
* Applying software engineering best practices to data management 

The Cube platform embodies software engineering best practices, empowering data engineers to [handle data models through code](https://cube.dev/docs/product/getting-started/databricks/create-data-model). This capability unlocks the potential to harness version control systems for secure data model evolution management, fostering collaborative efforts and streamlined code reviews. Additionally, it facilitates the integration of a semantic layer into the CI/CD pipeline and the creation of distinct development, staging, and production environments.

**Harness the capabilities and scalability offered by Databricks with Cube.** Cube seamlessly integrates with Databricks, enabling direct queries without the need for data extraction. This ensures that your data remains within the Databricks environment. Cube optimizes performance by leveraging the computing power of your Databricks instance, fully maximizing the scalability of your cloud data lakehouse. Additionally, you have the option to create [pre-aggregations](https://cube.dev/docs/product/caching/getting-started-pre-aggregations) within Cube, reducing the need for small, repetitive queries on your Databricks instance and allowing for background aggregate updates instead. Our aggregate-aware pre-aggregations are fundamentally different from common table or view materializing strategies often employed for performance reasons. With Cube pre-aggregations, users do not need to be aware of objects materialized for performance reasons; instead, they simply issue their queries, and Cube automatically chooses the optimal way to answer, from cache, from a pre-aggregate, or the source.

**Use your Databricks Lakehouse data in any application.** Cube supports your current and future use cases with first-class application programming interfaces (APIs). In addition to our SQL API, Cube also speaks REST and GraphQL, which are often preferred by application developers who are working with common front-end frameworks to develop embedded analytic applications. With Cube, you can take your investment in your Databricks ecosystem and build anything you can dream of.

## A step-by-step guide to building semantic layer with Cube and Databricks

In the below section, we’ll cover a quickstart step-by-step tutorial on how to start with Cube and Databricks. Remember: Cube Cloud is free to start (there is a sign-up button, up and to the right^). No credit card or calls with salespeople are required! :) 

Find a detailed getting started with [Cube Cloud and Databricks](https://cube.dev/docs/product/getting-started/databricks) guide in Cube documentation. 

### Create Databricks and Cube accounts
First, create Cube Cloud and Databricks accounts.

If you don’t have a dataset that you would like to use, you can follow [this guide](https://cube.dev/docs/product/getting-started/databricks/load-data) to create a test dataset in your Databricks instance.

### Connect Databricks to Cube

Please follow these steps to create your first deployment in Cube Cloud and connect it to your Databricks instance.

1. After you [sign in to your Cube Cloud account](https://cubecloud.dev/auth), click “Create Deployment”.

2. Give the deployment a name, select the cloud provider and region of your choice, and click “Next”.

3. Next, click Create to create a new project from scratch.

4. Select Databricks from the list of supported data sources and enter your Databricks credentials to connect to it. Get a [free Databricks account](https://www.databricks.com/try-databricks?utm=cube#account)

5. Once Databricks is connected, Cube can generate a basic data model from your data warehouse schema, which helps you get started with data modeling faster. We'll inspect these generated files in the next section and start changing them.

### Create your data model
Cube follows a dataset-oriented data modeling approach, which is inspired by and expands upon dimensional modeling. 

When building a [data mode](https://cube.dev/docs/product/getting-started/databricks/create-data-model) in Cube, you work with two dataset-centric objects: cubes and views. Cubes usually represent business entities such as customers, line items, and orders. In cubes, you define all the calculations within the measures and dimensions of these entities. 

To begin building your data model, click on “Enter Development Mode” in Cube Cloud. This will take you to your personal developer space.

Once you are in developer mode, navigate to the “Data Model” and click on any file located in the left sidebar to open it.

You can follow [this detailed guide](https://cube.dev/docs/product/getting-started/databricks/create-data-model) on how to make changes to your data model and build new measures and dimensions.

When you are ready to test updates to your data model, you can navigate to [Cube’s Playground](https://cube.dev/docs/product/workspace/playground). The Playground is a web-based tool that allows you to query your data without connecting any tools or writing any code.

### Query your data
You can query Cube using a BI or visualization tool through the Cube SQL API. For a better experience, we recommend mapping the BI's data model to Cube's semantic layer. This can be done automatically with Semantic Layer Sync or manually.

**Semantic Layer Sync**
[Semantic Layer Sync](https://cube.dev/docs/product/getting-started/databricks/query-from-bi) will synchronize all public cubes and views with connected BI tools.
1. Create a new sync by navigating to the “Semantic Layer Sync” tab on the “BI Integrations” page and clicking “+ Create Sync”. 
2. Follow the steps in the wizard to create a sync
3. Replace the fields for user, password, and URL with your BI tool credentials, then click on “Save All”. You can now go to the “BI Integrations” page and trigger the synchronization of your newly created semantic layer.
4. After running the sync, navigate to your BI tool instance. You should see the orders_view dataset that was created in the tool. Cube automatically maps all metrics and dimensions in the BI tool to measures and dimensions in the Cube data model.

**Manual setup**
Alternatively, you can connect to Cube and create all the mappings manually. 
1. You can find the credentials to connect to Cube on the “BI Integrations” page under the “SQL API Connection” tab.
2. After connecting, create a new dataset in the BI tool and select "orders_view" as a table.
3. Now you can map the BI tool metrics and columns to Cube's measures and dimensions.

That was a high level preview. You can create both you Databricks and Cube instances for free to try it out in your own environment.

Great work!

### More resources

Recorded workshop [Getting Started with Cube on Databricks](https://cube.dev/events/unlock-data-cube-snowflake)

Detail on [connecting to Databricks](https://cube.dev/docs/product/getting-started/databricks)


Deep Dives

Getting Started with a Databricks Semantic Layer

Over the last few weeks, we've announced updates to a few [Cube Cloud](https://cube.dev/product/why-cube-cloud) features that we've launched [earlier in June 2023](https://cube.dev/blog/cube-cloud-update-semantic-layer-sync-data-graph-saml). *Semantic Layer Sync* has expanded its suite of supported BIs with [Tableau](https://www.tableau.com) and *Data Graph* now visualizes [views](https://cube.dev/docs/product/data-modeling/reference/view) alongside [cubes](https://cube.dev/docs/product/data-modeling/reference/cube) to help you get a human-friendly overview of the data model within your semantic layer.

Additionally, we've released *Budgets* to help Cube Cloud customers with [annual plans](https://cube.dev/pricing) stay on top of your Cube Cloud usage.

## Tableau support in Semantic Layer Sync

[Semantic Layer Sync](https://cube.dev/blog/introducing-semantic-layer-sync) makes using Cube with BI tools as easy as possible. With the [latest update](https://app.contentful.com/spaces/6v81z8q9p5kl/entries/qEUh67oCvTTSqPAgB1lVv), Cube Cloud automatically populates and updates data sources within Tableau so you can start building workbooks and dashboards right away.

See how that works:

<video width="100%" controls><source src="https://ucarecdn.com/16a984fa-b3a2-411d-872d-184bc476f46a/video.mp4" type="video/mp4"/></video>

[Check the docs](https://cube.dev/docs/semantic-layer-sync) for details and try it today with your Tableau instance. Semantic Layer Sync with Tableau is available in Cube Cloud on [Premium and above tiers](https://cube.dev/pricing).

## Views support in Data Graph

[Data Graph](https://cube.dev/blog/introducing-data-graph) provides a human-friendly, visual way to explore contents of the semantic layer. It represents the data model as an entity relationship diagram and uses the crow’s-foot notation to mark join relationships between cubes. Data Graph is the best way to familiarize for you and your team to take a bird’s-eye view of the data model.

![](https://ucarecdn.com/bfc3e04a-b690-40bc-a6f8-14a9175fb4fd/)

With the [latest update](https://cube.dev/blog/complementing-data-graph-with-views), Data Graph is capable of visualizing not only [cubes](https://cube.dev/docs/schema/reference/cube) and [join](https://cube.dev/docs/schema/fundamentals/joins) between them, but also [views](https://cube.dev/docs/schema/reference/view) that incapsulate cubes and joins and serve as the front, public-facing layer of your data model.

See Data Graph with views in action and [check the docs](https://cube.dev/docs/cloud/workspace/cube-ide#data-graph) for details:

<video width="100%" controls><source src="https://ucarecdn.com/f4b0f37c-4a46-49fb-865f-7da96a093544/datagraphviews.mp4" type="video/mp4"/></video>

A quick reminder: Data Graph is available to Cube Cloud users on all tiers, so please feel free to check it out.

## Budgets

[Budgets](https://cube.dev/docs/product/workspace/budgets) help you stay on top of your Cube Cloud usage.

With Budgets, you can set up monthly or daily budgets for the Cube Cloud usage and get email notifications when certain thresholds of these budgets are exceeded.

For example, if you have an annual contract worth $12,000, you can set up a monthly budget of $1,000 with thresholds of 70%, 100%, and 120%. Later, you'll get email notifications when any of these thresholds are hit. You can also set up a daily budget to protect against Black Friday-like usage spikes.

<video width="100%" controls><source src="https://ucarecdn.com/038cb7bf-babc-4592-9692-9c3a1ee05449/video.mp4" type="video/mp4"/></video>

Budgets are available to Cube Cloud users on Premium, Enterprise, and Enterprise Premier plans.

## What’s next in Cube Cloud

While we're working on the next batch of Cube Cloud features, let me remind you that we're always happy to hear your feedback. Please don't hesitate to [get in touch](https://cube.dev/contact) or reach out in our [Slack community](https://slack.cube.dev) of more than 9,000 data practitioners.

Changelog

Updates to the newest features and tools to control your Cube Cloud usage

Cube Cloud update — Tableau in Semantic Layer Sync, Views in Data Graph, and Budgets

Three months ago, we've introduced [Semantic Layer Sync](https://cube.dev/blog/introducing-semantic-layer-sync) to completely bridge the gap between the semantic layer and BI tools and enable instantaneous, zero-friction self-serve analytics. We've launched with several [supported BI tools](https://cube.dev/docs/product/workspace/semantic-layer-sync#supported-tools) and promised to gradually expand the spectrum of BI tools that work with Semantic Layer Sync.

**Today, we're delivering on that promise and introducing the Tableau support in Semantic Layer Sync.** Now you can author the data model in Cube Cloud and have it seamlessly synchronized with [Tableau Cloud](https://www.tableau.com/products/cloud-bi) or [Tableau Desktop](https://www.tableau.com/products/desktop) where you can access the data from the semantic layer to build workbooks and dashboards.

In the following demo video, you can see that it takes seconds to configure Semantic Layer Sync and surface a [view](https://cube.dev/docs/product/data-modeling/reference/view) from Cube's data model to a workbook both in Tableau Cloud and Tableau Desktop:

<video width="100%" controls><source src="https://ucarecdn.com/16a984fa-b3a2-411d-872d-184bc476f46a/video.mp4" type="video/mp4"/></video>

Over the last months, Semantic Layer Sync has proven itself as a feature that saves a lot of time for data engineers who define metrics in the semantic layer and enable self-serve analytics for their end users in BI tools. Now, it's time to integrate Semantic Layer Sync in your Tableau workflow.

**Semantic Layer Sync with Tableau is available in Cube Cloud on [Premium and above tiers](https://cube.dev/pricing), and [you can try it today](https://cubecloud.dev/auth/signup).** Please check the [documentation](https://cube.dev/docs/product/workspace/semantic-layer-sync#tableau) to learn more about accessing and configuring this feature. Join our Slack community at [slack.cube.dev](https://slack.cube.dev) to share your thoughts and opinions about this update.

If you'd like to see Semantic Layer Sync in action, you can also watch the recordings of recent webinars where we demonstrated and discussed how Semantic layer Sync works with [Preset](https://event.on24.com/wcc/r/4243489/38E64F4551ABDA2967D79B9FFEAE8D3D) and [Metabase](https://event.on24.com/wcc/r/4321808/0C4A58D931932D9BBDD35D7FADB71776), other popular BI tools.


Tableau

Access data from the semantic layer to build workbooks and dashboards

Introducing Semantic Layer Sync with Tableau

For any cloud-based data vendor, your public cloud alliances are among the most complicated, and most important. In Cube’s case, as the universal semantic layer for building data applications, we deploy our product Cube Cloud on public cloud infrastructure and we connect to data sources and power applications built primarily on the public cloud. Recently, we have taken a major step forward with our partner Amazon by making them the first cloud marketplace that you can purchase Cube Cloud on. 

Cube Cloud now has a [public listing on the AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-cshbof57nzudw) which we could not be more excited about. Our customers have long had the ability to deploy Cube Cloud on AWS infrastructure, but during my tenure here we have been frequently asked about purchasing via AWS Marketplace. Customers want to do this for lots of reasons, but the most common are the ability to utilize their prior AWS committed spend, wanting a standard AWS EULA, or simply enjoying a more speedy procurement process. 

No matter the reason, we’re happy to be able to provide this option for our customers, should they want it. Flexibility and respecting our customers' right to choose the tools that best suit their needs is a core value of Cube, and we’re happy to extend this idea to the purchasing process as well. Cube customers who choose to purchase Cube Cloud via the AWS Marketplace will have access to all of the same editions of the software in the same configurations as they have when purchasing directly from Cube.

We are excited about taking this step to deepen our relationship with our partners at Amazon and I look forward to continuing to expand our integrations with AWS data and analytics technologies such as Amazon Redshift, Amazon Aurora, Amazon RDS, Amazon Athena and Amazon Quicksight. I would like to thank all of our partner contacts at Amazon who have helped us along the way with this partnership.

If you would like to learn more about how Cube Cloud can help you integrate all of your data sources into a single easy to manage data model and define consistent metrics accessible by all of your data applications then please [reach out](https://cube.dev/contact). We’re happy to help explore how you can build with the combined power of AWS and Cube.


A community for our customers and partners.

Cube Cloud Now Available on the AWS Marketplace

## Hard work pays off

When you’re at a growing company like Cube, everything is moving so fast that you don’t often take stock, slow down, and appreciate how far you’ve come. But this week has been a moment to do just that as we were honored to be recognized as a [Leader in GigaOm’s Sonar for Semantic Layers and Metrics Stores](https://cube.dev/gigaom-rates-cube-as-a-leader-for-semantic-layers). And - as to be expected - a Fast-Mover too. We’ve been launching new features left and right - in the last year [support for YAML](https://cube.dev/blog/introducing-cube-support-for-yaml-data-modeling), the [SQL API](https://cube.dev/blog/expanded-bi-support), the [Data Graph](https://cube.dev/blog/introducing-data-graph), the [Orchestration API](https://cube.dev/blog/introducing-orchestration-api), [Semantic Layer Sync](https://cube.dev/blog/introducing-semantic-layer-sync), and [Integrations with Langchain](https://cube.dev/blog/introducing-the-langchain-integration) and [other AI tools (Delphi!)](https://cube.dev/blog/conversational-interface-for-semantic-layer) - so are excited to be recognized for all the hard work. 

## But why does it matter? Understanding the Significance of Semantic Layers

Semantic layers and metrics stores play a pivotal role in the world of data analysis. They act as bridges between raw data and actionable insights, enabling organizations to harness the full potential of their data. A semantic layer, in particular, consolidates complex data into a format that is understandable across different teams and tools, effectively translating raw data into common business terms.

## Why Cube Stands Out

GigaOm noted several compelling reasons that made Cube standout: 

**Code-First Orientation**: Cube's [strong code-first orientation](https://cube.dev/docs/product/introduction#code-first) is a distinguishing feature. We firmly believe that the future of data engineering lies in applying proven software engineering practices - such as version control, code reviews, infrastructure testing -  to data management.  This will lead not only to data engineers having very granular control over the semantics of their data, but also structure these data models for reusability and maintainability. However, we also know that being able to work with data models with a GUI (graphical user interface) is also sometimes preferred. That’s why we combine the best of both worlds to allow data engineers to be code-first but the ability to jump into a GUI-based experience - like our recently launched [Data Graph](https://cube.dev/blog/introducing-data-graph). We are happy that GigaOm agrees.  

**All the APIs**: Cube's native API support ensures seamless integration with various data sources and analytics tools, making it a versatile choice for organizations with diverse tech stacks. We offer both data APIs -  - GraphQL, REST API, and SQL API - as well as the Orchestration APIs and monitoring integrations to not only send modeled data to whatever data experience your users want, but also integrate Cube monitoring and observability into all the usual DevOps tools.  

**Analytics Pre-Processing**: Cube's ability to perform analytics pre-processing through caching and pre-aggregations is a significant advantage that many competitors miss. This is especially critical in embedded analytics use cases where a data engineer is creating analytics for their customers. While it might be OK for an employee to wait for a minute or two when generating insights, it is never OK for a customer to wait even a nanosecond longer than necessary. And with Cube, customers can speed up queries from minutes to seconds - and seconds to sub-seconds - even with large datasets.

**Check out our position on their sonar report.**
![](https://ucarecdn.com/d266a78b-3dcf-4355-a59d-17a15007afa1/) 

**Yes. We’re a Platform, Not Just a Feature**
We love that Gigaom recognized us as a platform play vs. just a feature, as we believe that having a semantic layer will become a necessity for all data stacks in the very near future. Only a company that has built a platform and is able to integrate with any data source and any data experience - customer dashboards, BI tools, or even AI agents - will be able to serve as that missing piece of the data stack. 

**Beating the Big Brands**
Part of the advantage of having an open source version of Cube is that we have been able to grow faster with thousands of developers reviewing, adding, and using our code. How else were we able to leap ahead of SAP, Oracle, and even Google!? 

### What Else Might you Get from this Report

But if you want more than a glowing evaluation of Cube, you can find it in this report. GigaOm has provided plenty of information about semantic layers and how they fit into various organizations both big and small as well as real-world use cases to help drive home the need for one. Learn what to consider when you decide to adopt a semantic layer, how to make that decision, and how to implement it into your data stack with minimal interruptions.

### Get the Full Report and See for Yourself

For a comprehensive look at Cube's performance and a deeper dive into the world of semantic layers and metrics stores, we encourage you to read the complete GigaOm Sonar report. We [offer the report here for free](https://cube.dev/gigaom-rates-cube-as-a-leader-for-semantic-layers).

Jen Grant

Hard Work Pays Off 

Cube recognized as a Leader and Fast-Mover in GigaOm’s Sonar for Semantic Layers

Data pipelines, data modeling, and BI tools to collect, store, and analyze data at scale.

Front-end applications and low-code tools to bring data insights to end users.

Application Development

Company news, product updates, and engineering deep dives.

Latest news and insights from Cube

Data Engineering

Application Development

All Things Cube

Semantic Layers are the missing piece for AI-Enabled Analytics

Building your data stack: Cube Cloud or OSS

The Need for an Open Standard for the Semantic Layer

Introducing a tool for Looker to Cube migration

Introducing dbt integration with Cube

Get Cube updates to your inbox

Introducing Python and Jinja for data modeling

Introducing Python for configuration

Getting Started with a Databricks Semantic Layer

Cube Cloud update — Tableau in Semantic Layer Sync, Views in Data Graph, and Budgets

Upgrade your data stack today

Introducing Semantic Layer Sync with Tableau

Cube Cloud Now Available on the AWS Marketplace

Cube recognized as a Leader and Fast-Mover in GigaOm’s Sonar for Semantic Layers

Data Engineering

Application Development

All Things Cube