At Databricks, nothing makes us happier than making our customers extra productive, which is why we’re delighted to announce a local adapter for dbt. It’s now simpler than ever to develop sturdy information pipelines on Databricks utilizing SQL.

dbt is a well-liked open supply instrument that lets a brand new breed of ‘analytics engineer’ construct information pipelines utilizing easy SQL. The whole lot is organized inside directories, as plain textual content, making model management, deployment, and testability easy.

With the brand new devoted dbt-databricks adapter accessible in public preview right now, dbt builders can get began by merely working pip set up dbt-databricks. This bundle is open supply, and constructed on the sensible work led by dbt Labs and the opposite contributors who made dbt-spark attainable. Not solely did we streamline the set up by eradicating any dependency on ODBC drivers, we embraced dbt’s “conference over configuration” for optimum efficiency:

• dbt fashions use the Delta format by default
• Incremental fashions at all times leverage Delta Lake’s MERGE assertion
• Costly queries like distinctive key era are actually accelerated with Photon

Extra enhancements to this adapter are coming as we proceed to enhance the general integration between dbt and the Databricks Lakehouse Platform. With record-breaking efficiency and full assist for traditional SQL, it’s the finest place to run information warehousing workloads, together with information pipelines constructed with dbt.

We’re additionally excited in regards to the upcoming addition of dbt Cloud to Companion Join, Databricks’ one-stop store for its prospects to find and combine the very best information and AI instruments in the marketplace. dbt Cloud is a hosted service made by dbt Labs, which helps information analysts and information engineers collaboratively construct and productionize dbt tasks. Coming in January, any Databricks buyer will have the ability to begin a free trial of dbt Cloud from Companion Join and routinely combine the 2 merchandise. That mentioned, the 2 merchandise already work nice collectively, and we encourage you to join dbt Cloud to Databricks right now.

Talking of dbt Labs, we hope to see you at their convention, Coalesce, which begins right now! Reynold Xin will likely be having a fireplace chat with Drew Banin, CPO for dbt Labs and Ricardo Portillo will likely be talking about constructing information pipelines for Monetary Providers leveraging dbt and Databricks. You need to undoubtedly test it out and be part of the dialog on the dbt Group Slack in . We stay up for your suggestions!

Keep tuned for extra thrilling updates on how Databricks works with dbt and watch our Github repository for brand spanking new releases.