image: thumbnail.jpg
Databricks x Posit | Improved Productivity for your Data Teams
0:00-7:30 Introduction to Databricks
7:31-15:40 Introduction to Posit
15:41-27:51 Demo of VS Code, Model Serving
27:52-30:00 Demo of signing into Posit Workbench via OAuth
30:01-37:50 *New* Databricks pane: accessing Databricks compute and exploring data in Unity Catalog via Databricks Connect
37:51-43:40 *New* Simplified ODBC access to Databricks
43:41-50:06 Publishing data apps to Posit Connect with a Databricks backend
50:07-51:18 Demo recap
51:19-54:15 Roadmap and the future
54:16-1:07:33 Q&A
Talk with our team at Posit by scheduling a call here: https://pos.it/databricks-posit-event
Databricks and Posit are partnering to help professional data teams do more with the power of their favorite data science tools seamlessly integrated with the Databricks Lakehouse.
Our partnership aims to help Python and R developers build and share valuable data-driven content using familiar, state-of-the-art tools that maximize their productivity, all while leveraging the data processing and governance capabilities of Databricks.
In this joint event, James Blair, Product Manager of Cloud Integrations at Posit, and Rafi Kurlansik, Lead Product Specialist from Databricks, will be sharing a live demo covering how data teams can build and share data-driven content more easily with the joint power of Posit and Databricks, along with an exciting look into the future of the partnership.
Transcript#
This transcript was generated automatically and may contain errors.
All right, well, welcome, everyone. We're excited to see you all here. My name is James Blair, and I work as a product manager at Posit for cloud integrations. And I'm excited to be here with you all today to talk about some of the recent work that we've done with Databricks and the partnership that we have with them. And and what we're excited about. I'm joined today by Rafi Kurlantic, who is a product specialist at Databricks. And we'll be walking through some of the Databricks perspective on the partnership, as well as the work that we're doing together. So I'm actually going to turn the time over to him to get things started today. But we're excited to be here with you all.
OK, awesome. Thank you so much, James. So at Databricks, I'm going to go into a little bit of who we are and what we're all about. So our mission is to democratize data and AI to the entire world. And we our strategy that we use to basically accomplish this mission is twofold. First, we have created three very successful open source projects, Apache Spark, which is scalable in-memory data processing, Delta Lake, which is scalable data storage and management, and MLflow, which is used to manage the entire machine learning lifecycle. In addition to this commitment to open source, we also have developed a managed service that is beloved by many of our customers. We have over 6,000 employees across the world. We're very successful, thankfully, in terms of our revenue and investment from top firms around the world. And we're the inventor of the data lake house paradigm, as well as a pioneer in generative AI. And we're very fortunate to have been recognized for these accomplishments.
The data lakehouse and data intelligence platform
Now, what I would like to begin to go into is sort of, you know, our view on data and AI is that it's universal. And everyone already knows that this has impacted every single industry, every single sector of the economy. And we believe that the companies that kind of win in their industry will become themselves data and AI companies. This is easier said than done, though. And if you look at what data and AI means, it's actually a collection of very different but related technologies. So, for example, you have everything from data warehousing and AI to streaming, real-time analytics, data science, machine learning. And then you have to govern all these different sort of use cases and technologies. Not to mention that you have the emergence of generative AI and the promise that it kind of brings to the table. So it's very, very hard for organizations to kind of wrap their heads around all of this and make them, you know, successful with all of this.
Our effort has really been to simplify all of these things. And we've done so with the data lake house. So if you're not so familiar with the lake house, let me just kind of walk you through it quickly. So the concept of a lake house is essentially it's a portmanteau of a data warehouse and data lake. So you have the structured data and performance and consistency that you have with the data warehouse alongside the unstructured data and sort of the open nature of a data lake. So our data lake house is built on an open data lake. Basically, you use cloud object storage, S3, ADLS, GCS to store all of your data. And then you can unify your data under one single storage and management system, which is Delta Lake is the underlying technology for that. Once you have all of your data in storage and you're managing it with Delta Lake, then you can secure and govern it with Unity Catalog. And then once you've secured and governed your data, all of your data, one copy of the data, then you can kind of unlock these different pillars of data and AI and analytics.
So we kind of pioneered this concept in 2020, and we're really, really excited and happy to announce that this has been pretty successful. We've had 74% of global enterprises have adopted the lake house architecture. So this really speaks to kind of the power of it and sort of the simplicity and unification of it.
All right, now that you understand a little bit about what a data lake house is, it's this open, unified foundation for all of your data. What happens when you combine this type of foundation with generative AI? So how does that change things? We think that this actually introduces a new type of data platform, and that is the data intelligence platform. The concept of a data intelligence platform really does further the democratization of data and AI. It makes it more accessible to more people than ever before.
So practically, how does that work out? What does this really mean practically? So I want to touch on two elements that this really kind of means. The first is that obviously you want to be able to use generative AI to understand the semantics of your data so that you can ask questions of your data in natural language. That's a huge part of what a data intelligence platform is, and we are building Databricks to support that type of interaction with data. The other way that you can kind of use generative AI is to use open source models and combine it with your own personal data so that you can build applications on top of it.
So that's sort of what I'm going to focus on a little bit more today when we go through a demo. But rest assured that Databricks is working on these two different sort of components of a data intelligence platform, both for end users who want to ask questions of their data and for developers that want to build with generative AI.
Databricks products for Posit users
Okay, so how do you access this data intelligence platform using Posit products? So James is going to focus a lot on this question, and I'm going to show a little bit as well. But I want to talk really quickly before I hand it over to James. I want to just talk quickly about the products that I'm going to be using from Databricks that enable this type of activity.
So the first is going to be Databricks Connect version 2. Databricks Connect is a lightweight client library that you can install on your laptop or any server that you want to be running a remote connection to Databricks. And it allows you to use Apache Spark in a very simple way. So it's very, very useful when you're working in your IDE and you want to kind of do exploratory work or development work in the comfort of your IDE. But it's also very useful when you want to develop a data app like a Shiny app or Streamlit app or something like that. And you want to be able to access Databricks and all the data that's in there, that's secure, that's governed, all that kind of stuff. So that's the first thing I'm going to be showing you today.
The second thing I'm going to be showing you is the Databricks extension for VS Code. I'm going to show you how easy it is to get set up, how you can find it in the VS Code marketplace. How it kind of lets you use all the features that you like about VS Code, but still be able to access and run code against Databricks and work with the data that you have there.
And then lastly, I'm going to be showing a little bit about Databricks Model Serving. This is our online real-time inference for machine learning models. It is fully managed, including support for GPUs. It's one-click deployment of models. It's serverless. It scales up and down seamlessly. We've done special optimizations specifically for serving large language models. And you can also set up monitoring of the inference. You can set up tables that basically collect the inferences that you're making and help you set up monitoring sort of just like out of the box with a few clicks.
Introduction to Posit
Okay, now I'm going to turn it over to you, James. Hey, thanks, Rafi. I think it's clear that Databricks has a lot to offer and a lot to bring to the table with this data intelligence platform. And we're excited to talk about the ways in which Posit has worked closely with Databricks to make the experience for our joint customers even better. To highlight this, I want to talk a little bit about the history of Posit as a company, which may be familiar to some of you, it may not be familiar to some of you. So just to give some context of where Posit comes from and how we fit into this picture, I think will be good.
Okay, so Posit has its roots in open source software and primarily in the form of the RStudio IDE. So if you've ever used the R language, odds are at some point you've used the RStudio IDE, which is where Posit has come from, right? Historically, we continue to be the maintainers of this open source IDE today. And it is the preferred development environment for developers and data scientists that are working with the R language. We also contribute heavily to the open source community through a variety of open source packages. These include a number of popular R packages like the tidyverse and its collection of packages for data analysis and data science, Shiny for interactive applications, Quarto for dashboarding and for reproducible reporting, Reticulate allows R users to work with Python directly from the R language, and many, many more.
One of the things that's exciting as we look at where Posit has been and where we're going is our renewed and continued investment into not only R but also Python, so that we focus not just exclusively on a single language, but instead we focus on promoting open source technology for data scientists, regardless of whatever language that might be in.
Just a little bit of historical context, and this is not comprehensive at all, but just to give some context as to what Posit is as a company and who we are. In 2011 is when the RStudio IDE was first introduced, again, as an open source tool that's available to download, and it continues to exist today as an open source desktop installation that anybody can go and use. That's something that we firmly believe in is providing those open source tools to give anyone in the world access to the tools necessary to be data literate and to make informed decisions based on data they have access to. Shiny, the package for building interactive web applications in R was introduced in 2012. Our first enterprise software where we were charging customers to run software in an enterprise environment with additional features that were attractive to the enterprise were introduced in 2014. And then in 2020, we announced that we were becoming a public benefit corporation, and included in that effort is some specific promises that we make to our community as part of our company mission. And then in 2022, the company rebranded from RStudio, which had been the name of the company up until that point, to Posit. So if you've ever worked with RStudio, the company, we are one in the same. We just now go by the name of Posit, which helps us, again, kind of push forward our mission of open source data science, regardless of the language of choice.
One thing that's pretty unique about how Posit operates is that a lot of what we do today on the engineering side is focused on open source technology and open source packages. Again, we continue to support the open source RStudio IDE that continues to be very, very popular amongst data scientists and users. But then we also have this growing collection of packages that we work to support as well. That we firmly believe that this open source commitment is beneficial across the board. It helps users get early access to tools and technology that enable them to be successful in working with data at no financial cost. And then it also enables them to take those tools into a professional context, into a commercial environment, where we can support those tools with our enterprise products that customers purchase. And then that purchase, that income that we receive there, gets reinvested back into the open source side. And this is something that we refer to often as the virtuous cycle. We give to the open source side, and that eventually promotes commercial adoption of the tools that we provide.
And this is something that we refer to often as the virtuous cycle. We give to the open source side, and that eventually promotes commercial adoption of the tools that we provide.
When we talk about the commercial side or the enterprise side of Posit, there's really three main components that we'll talk about. Each of these will be highlighted to some extent in our conversation today as we talk about Databricks. But just to give those who may be unfamiliar an idea, we offer a tool called Posit Workbench, which enables users to access common data science development environments through a browser in a centrally managed type of way. So through Posit Workbench, you have access to the RStudio development environment, VS Code, Jupyter Notebook, JupyterLab. And there's a number of different ways that Posit Workbench can be configured to run within an organization and can be designed architecturally to fit within whatever type of environment an organization is running in.
As a complement to Posit Workbench, we offer Posit Connect, which serves as a bridge between the developers and the data scientists and the end users. So Posit Connect allows developers to publish and share the work that they've done inside of a development environment, whether that work is something interactive, like a Shiny application that was built in R, or a Streamlit application that was built in Python, or even a web API that was built in R or Python. Those can be published to Posit Connect. And then other users of the organization, whether they're business users, decision makers, they may be downstream processes that are consuming APIs. Those can then utilize the work that's been done by the data scientists and the developers, because it's hosted and shared in a central location that's secured and governed according to the rules of the business and the rules of the organization.
As a kind of behind the scenes component to all this, we also offer Posit Package Manager, which enables organizations to provide a secure and essentially regulated way to have users access R and Python packages. Organizations can define which packages are part of their ecosystem. Posit Package Manager supports snapshotting by date. So if I had an analysis or a thing that I worked on two years ago, and I need to roll back to packages as they were at that point in time, Posit Package Manager makes all that very simple, very straightforward, and is a benefit to organizations where reproducibility and security are of most importance.
So what we'll talk about now as we move forward today is how all of these different pieces, the open source side, the commercial side, how does all this work together with Databricks? And what is it that we're doing today? And what is it that we plan to do in the future to make life easier for those organizations that happen to be using Posit and Databricks together? Collectively, we refer to the commercial side of the commercial products that we offer as Posit Team. And again, that idea of Posit Team is Posit Workbench, Posit Connect, Posit Package Manager. And what we look forward to and what we're going to talk through today is how Posit Team can sit alongside this data intelligence platform that Rafi's described and interact through a variety of different ways with that platform and leverage all the Databricks has to offer in terms of data governance. Governance, security, compute power, distributed computing, all the Databricks has to bring to the table, model serving, all these things that Rafi's touched on already.
We can leverage those and make use of them from inside of these different products that Posit offers. And we can do that through open source packages. We can do that through specific things that we've introduced into the RStudio IDE that we'll highlight in just a moment. But the idea is Posit Team can work directly with tools and functionality of the Databricks data intelligence platform today and in the future. And we're excited to show a little bit of what that looks like.
Awesome. Thank you, James. All right. So I'm going to go through a quick demo here, and I want to start with Posit Workbench. So for people who may not be so familiar, you sign into Posit Workbench, and then you're able to basically launch different sessions for different development environments that you would like to use. So you can create a new Jupyter notebook session, JupyterLab, RStudio if you want to develop in R, and then, of course, VS Code, which is preferred by many for Python as well as Jupyter.
Now, I've already gone ahead and created a VS Code session. But if I wanted to, obviously, we could create more, and I could have multiple different ideas going on at the same time. My VS Code session is over here, and I want to start off by just showing how we can get access to the VS Code extension.
Okay. So from VS Code, I can come over here to the extensions tab, and then if you just search for Databricks in the marketplace, you'll find the very first one is owned by the Databricks organization. You can install it, and then you're good to go. So it's that simple. Once you have the extension installed, you'll see this icon over here in the left nav bar. And the first time that you open this up, you'll basically be prompted to configure it, which will ask you, you know, which Databricks workspace do you want to sign into? So you can specify that URL for your particular workspace here.
Once you choose that, you have the option of signing in via OAuth, or you can use the Databricks CLI. I've already gone ahead and installed the Databricks CLI here. Actually, James has done that on my behalf. And I've configured and I've logged into Databricks through the CLI as well. Point being that it's very, very simple to sign in to Databricks through the UI just by clicking. You don't have to go through and mess with environment variables if you don't want to.
And once you do that, once you set up the connection to the Databricks workspace, you'll see this left page over here basically, or left window basically be populated. So we'll see what workspace I'm connected to, which cluster in the workspace I'm connected to. I can actually browse the various clusters that I may want to connect to and choose which one I want to attach my session to. And this is going to allow us to basically use Databricks Connect and run files holistically on this particular compute resource that lives in Databricks.
The last thing I'm going to cover, I'm not going to do like an exhaustive sort of view of the VS Code extension because of time constraints. But just to call it out, you can also choose a destination in the Databricks workspace where you can sync your code to. So this is very, very helpful if you want to have relative paths work when you run code on Databricks from VS Code. So you basically have a copy of your code in Databricks so that all the relative paths will resolve correctly. But it's also helpful if you want to do some kinds of development work in VS Code, and then you want to maybe flip to Databricks to the notebook to kind of use some of the features that we have there as well. So you have an exact sort of high fidelity copy of your code in Databricks too.
Demo: Databricks Connect and UFO sightings data
All right, so I'm going to start off. I've already gone ahead and connected to this particular compute. And I'm going to go through a Jupyter notebook that I have showing how we can use Databricks Connect. All right, so the first thing that we're going to do, just remind everyone about Databricks Connect. This is essentially a library that is installed on this Posit Workbench server where this is running in the cloud. Outside of Databricks, this is a remote connection. And normally we just create a new Spark session. And then any PySpark code that we use, any Spark code that we use in the DataFrame API will be run remotely on Databricks.
If we were to run any local or any just pure Python code, that would still be running on the machine where this is installed. So it's important to just keep that in mind that Databricks Connect allows you to use Apache Spark to process data in the cloud. But anything that's not using Apache Spark is going to be executed on the local machine.
Okay, now one of the nice things about the VS Code extension is that it has a particular integration with Databricks Connect. Where if you're using the Jupyter notebook in VS Code, it will actually just create the Spark session for you. So here you'll see I'm going to read from a UFO sightings table that is in my catalog under the default schema and then the UFO sightings table. Okay, so I'm accessing data that's in Unity Catalog here. I'm bringing it to Pandas because I just want it to render a little bit more nicely. And this is a very interesting data set because it's essentially records of UFO sightings in the United States.
You can see which particular country, which state, what time it was. And then the thing that I want to focus on actually is the text. So the text is the actual report itself, like what did someone observe. So I live in New Jersey, so I'm going to filter down to the state of New Jersey and we're just going to take a look at the text of one of these sightings. Okay, so we have an object like a plane without wings and so on and so forth. There's a lot of descriptive text here.
It would be a little bit challenging to kind of go through and read all of these reports and draft up an analysis of what are the similarities and what are the discrepancies between the various reports. I think this is a very, very good use case to basically take all this text data, give it to a language model and ask it to summarize it for us. So I'm going to show how we can do that with Databricks Model Serving.
I think this is a very, very good use case to basically take all this text data, give it to a language model and ask it to summarize it for us.
Databricks Model Serving
So the Model Serving product on Databricks is remarkable because it allows you to essentially serve via REST API every type of machine learning model that you would want. So everything from classic machine learning, everything that we were so invested in up until the rise of generative AI, you can serve all of those on Databricks. This is going to be done in a secure and scalable way. And this also applies to foundation models. So we do the work to go get the best state of the art open source models so you don't have to go do all that research. And we'll actually have those hosted in your environment for you. And you'll be able to call those very simple, very simply.
And then third is that we can actually provide a secure route or gateway that allows you to manage if you want to use commercial offerings from like open AI or other companies. You want to manage that under the same place that you manage your custom models or foundation open source models. We provide the means for you to do that as well. So we kind of bring all these different kinds of machine learning models into a single serving product.
Now, foundation models in particular, these are the state of the art open source models. They're really good for summarization, for content generation. If you want to build a chatbot using RAG or resource augmented generation. And just FYI, this is per token based pricing. The foundation models API is in preview. If you're interested in this kind of thing, you like what we're showing you today, you should definitely reach out to your Databricks rep.
So what I'm going to do here is I'm going to take just eight observations from the data frame that we created before. And we're just going to take the text for those eight observations. So here is these UFO sightings from New Jersey. We have eight observations here. Now what I want to do is I want to access a Llama270b chat model that we have hosted in Databricks. And just call it from VS Code right here. So I'm going to install a, I've already gone ahead and installed this. I'm not going to run this right now. But we're going to install this inference SDK.
And then let's start this here. We'll import this chat completion. And we'll define the prompt, which is going to be, you're a helpful assistant. You're going to summarize UFO sightings and explain the similarities and differences between them. Summarize them in three bullet points. We want this to be pretty snappy and pretty concise. And then to actually make that call, we just have several lines of code. Very, very simple. Where we pass in the system prompt and we pass in the sightings object that we created before, that blob of text. And then here's our response.
So we have three bullet points. It says all the sightings describe objects or lights in the sky that can't be identified as man-made or natural. Many of them describe as moving quickly or hovering. And some describe as disappearing or changing direction suddenly. And let's take a look at some of the differences. Some of the sightings describe, oops, some of the sightings describe the objects as being very large. While others describe them as being relatively small and so on. So I think this is a really, really nice way to kind of combine Databricks model serving with a challenging data set. And leverage some of the superpowers that LLMs actually have.
One last thing I want to call out here before I move on is notice how I didn't have to specify any credentials or anything like that when making these API calls. So all of the Databricks SDKs in the last year that have come out and they've been designed to reuse credentials that are created when you sign in with OAuth or when you use the Databricks CLI. So this is a much more secure and easy way to use these SDKs.
Now the second sort of example that I want to show you is using the Python SDK. So Databricks recently released a pure Python SDK for interacting with the Databricks REST API. And I want to show you how simple it is to call a language model on Databricks using the Python SDK. So we'll import the workspace client from the Databricks Python SDK. We'll instantiate the workspace client. We're going to say what is the serving endpoint, model serving endpoint that we're going to call. And then we'll say our prompt.
In this case, we're not looking at the UFO data anymore. We're going to be calling a model that actually has been built on top of a RAG architecture that indexes all the latest Databricks documentation, stores it in a vector database, and then uses those embeddings to actually return a more accurate response to us when we query. Querying it is also super simple. So we use the workspace client, the serving endpoint service, and then we pass in the query, which is go hit this model. And then here's our prompt that we want to pass. And for this one, I'm going to use the run Python file functionality here in VS Code.
And we'll just wait a few seconds. We should be able to see the results here. Okay, so what does our language model have to say about this? Databricks model serving deploys your MLflow machine learning models and exposes them as REST API endpoints. Serverless compute resources, easy configuration, high availability and scalability, dashboards, monitoring, all sounds pretty great.
Okay, now I'm just going to show quickly and I'm going to hand it back over to James. So this is in Databricks itself. This is the model that we were calling. And you can see some of the dashboards that exist here, right? Beyond the scope of the presentation to go into detail here, but I just wanted to show how this is the model that we were calling there.
Okay, so to recap, we showed how you can launch VS Code in Posit Workbench, how you can access Databricks by installing the VS Code extension. You can use Databricks Connect to have interactive sessions with the data that's being managed in Databricks and Unity Catalog. And then we also showed how you can access Databricks model serving both through the Python SDK and then also through that Foundations Models API. All right, and now I will pass it back over to you, James.
New Posit Workbench features for Databricks
Excellent. Thanks, Rafi, for walking us through that demo. I think, again, it's clear that Databricks has a lot to offer in regards to the ways in which we interact with modern tools in the data stack. And it's really great to see some of those become more and more available to users from whatever environment they want to be in, whether it's VS Code or something else. And we're going to continue that theme here as we talk about some new changes that we're introducing into Posit Workbench.
So a lot of what we're going to walk through here in a moment is going to be made available in the upcoming December release of Posit Workbench. If you happen to be a Posit Workbench customer. And there's also some open source components here that we'll talk through as well.
The first thing I want to highlight is when I'm in Posit Workbench, if I go to start a new session, I'm presented with this option to choose, like Rafi demonstrated earlier, which editor, which environment I want to run my session in, whether it's Jupyter Notebook or Lab or VS Code, whatever the case might be. But there's this new portion down below the session name where I can choose a Databricks workspace that I'm going to want to authenticate into. These workspaces are configured by an admin. So an admin would go through and define what workspaces users would have access to. And then I can select from whichever workspace I want to select from, whether it's none or a predefined one, and then choose to sign into that workspace.
Once I click sign in, this will transfer me over to another browser window where I will sign into Databricks using whatever mechanism is used within my organization. On our side, we use SSO. And so I would go through that whole flow to get signed into Databricks. Once that's happened, Posit Workbench will manage a specific token credential for me that I won't need to worry about. And the nice thing about that and the benefit of this is, as a user, I don't need to worry about downloading or managing a personal access token to gain access to Databricks from within Posit Workbench. It's all managed behind the scenes for me and automatically kept up to date once I've gone through this sign in process here from the new session launch screen.
Once we've started this session, we'll be dropped into the RStudio IDE. So if you've used the open source version, the version that's available on Posit Workbench feels essentially identical and familiar.
I have the ability to browse files and packages and other things that I might find interesting in here. I'm going to go ahead and open a project that contains some content that we're going to walk through as part of this demonstration.
And then I want to highlight one of the things that's going to be new in this upcoming release of Posit Workbench. If I look up here in the top right hand corner, I have a number of different panes that I can open up. There's a new pane, however, called Databricks. And if I click on this new pane, this will open up a view into my Databricks compute console.
So essentially what I am able to do here is I'm able to see what compute clusters I have access to on Databricks. I can see what status they have, whether they're running clusters, whether they're stopped, whether they're in the process of starting up or shutting down. And if I want to, I can click on any of these. So let's take a look at this old cluster here at the top of my list. I can click on any of these and find additional details about the cluster, including its ID, what its policy is, information about the resources assigned to that cluster.
All of this is made available directly here inside of the RStudio IDE within Posit Workbench. And again, this is specific to Posit Workbench. So to be transparent, this is not something that will come to the open source version of the RStudio IDE, but rather is something that is specific to Posit Workbench.
The advantage of this is that it kind of twofold. One is as a Databricks user, if I want to run some workload on Databricks, I can start the cluster I want to run that workload against directly from within RStudio. I don't need to log into the Databricks UI. I don't need to navigate some other system. I can do everything I need to directly from within the development environment window.
And then once I've started a cluster, so let's take this old cluster, for example. Once I've started a cluster, I can easily connect to it by clicking on this connection icon. This will open up a dialog box that pre-populates all the information that's necessary to make the connection to this cluster.
And again, one of the advantages of the fact that Posit Workbench is managing my Databricks credentials for me is that I don't need to input any sort of password. I don't need to input any sort of key or token here. All of that is automatically passed through from Workbench to the connection when this connection is made. You'll see here that this cluster ID was automatically populated in this connection dialog. It tells me, okay, the Databricks runtime for this cluster is 13.3. And then it informs me here that the Python environment necessary for DBR 13.3 isn't available locally.
So Databricks Connect is something that Rafi talked about a little bit. And that's the tool that we use to connect remotely into these Databricks sessions or these Databricks compute clusters to orchestrate workloads there. But there are some local dependencies necessary in order to have that functionality. I need to have a version of Databricks Connect locally. And that version of Databricks Connect needs to align with the version of Databricks runtime on the cluster that I'm connecting to.
So all of these checks happen automatically. And so in this case, this is a cluster that I just started this morning for this purpose with an older version of Databricks runtime, so 13.3. And it tells me I don't have the right Python dependencies in place. I'll need to install those. So we'll go through that process in just a moment. It tells me down here, look at your credentials are being managed by Workbench. We've already passed the credential. Everything's good to go for connecting. The only thing missing is the local Python dependencies that I need to make this connection.
If I go ahead and click OK here, then this will launch down here inside of we did this right. Let's go back through here one more time. There we go. Let's copy this to the clipboard. OK. So if we run this code here to make the connection. I'm going to copy. We'll try this one more time.
All right. OK, so if we run this code here inside of our console in our studio, we'll get a prompting to install the right Python dependencies for this particular connection. So here in this case, it says, look, we don't have the right version of dependencies necessary to support a connection to a cluster running DBR at 13.3. Do you want to install those dependencies? And we can hit yes here. And this will go through the process of creating a virtual environment on my specific to my user, in this case, specific to my user on Posit Workbench that contains all the correct dependencies necessary to create a connection to this specific cluster.
Now, if I were to create another cluster or try to connect to another cluster that was also running a version of Databricks runtime, the same version of Databricks runtime. So in this case, 13.3, it would know that this environment already exists and use this preexisting virtual environment. So I don't need to recreate virtual environments every time I connect, but rather each time I connect to a new version of Databricks runtime, I'll be prompted to install an updated version of the Python dependencies. And those will be isolated in their own virtual environment so that I can reuse whichever one I need to for the clusters that I might want to access.
You'll notice up here on the top right hand corner now, once I've made this connection, it automatically opened to this connections pane. This connections pane now shows me a view into Unity catalog. So I can browse through the different catalogs, schemas and tables that I have available to me on the Databricks side and view specific details about those tables. So if I wanted to look at the UFO data that Rafi was talking about, I could come in and expand this out and see, okay, here's this New Fork reports data or this New Fork reports table. It's inside of this demos catalog and this New Fork schema. And then this will open up in just a moment and show me a preview of the different columns that that table has, the different values that that table has.
This is all everything that we've walked through here, with the exception of this Databricks pane over here on the right hand side. Everything that we've walked through here is a function of the sparkly R package, which is an open source package. So being able to connect to Databricks from a local R process using sparkly R is something that's available in that open source package. It's just this Databricks pane that's specific to the Posit Workbench release that's coming later this month.
That also includes just a highlight that also includes the Python dependency management that we talked about. So sparkly R will do a bunch of checks behind the scenes to ensure that everything is in place locally to facilitate the proper type of connection into a Databricks environment.
Connecting to Databricks and exploring data
Once I've created this connection, I'm actually going to go ahead and close out of this one here. And I want to walk through a little bit of an example of some of the things that we can do. So we can, there's two different ways that we can interact with or create these connections into Databricks to work with the data that's being stored there. One is through sparkly R, which we just showcased. And here's another example of how I would make that connection past the cluster ID. And then the key here is this method is Databricks connect. This is the argument that lets sparkly R know how we're trying to make this connection.
So I can go ahead and run this again. This will create a connection to a separate cluster. You'll notice this cluster is running 14.0. And I did not need to install any sort of Python dependencies because they're already in place in my environment. In fact, if we look at here, here, I can list the virtual environments that are available for me. And I can see that I have three different virtual environments specific to Databricks that are all specific to different versions of Databricks runtime. And again, sparkly R will help facilitate the automatic instantiation of these if necessary.
Simplified ODBC access to Databricks
In addition to connecting through sparkly R, we're also excited to announce that we will begin offering a Databricks odbc connector to our professional customers here in the next short little while. And this Databricks odbc, or yeah, this Databricks odbc driver will be made available alongside the collection of existing odbc drivers that we offer to customers.
What that means is if I come in and I say, let's go back to connections, let's create a new connection. Here you can see I have a long list of available connection types. And if I scroll, there we go. Down to the middle of this list, I can see that Databricks is listed in here as one of those drivers that I can connect to. It's also worth noting that this Databricks connect DBR 13 plus is this is a way to connect directly from the connections pane with the sparkly R package. So if I wanted to connect through sparkly R, I have a number of different ways that I can facilitate that one of which is through the connections pane, which will bring up this dialogue similar to what we saw previously.
If I go to the Databricks odbc driver here in this list, this will prompt me for the HTTP path that I need to connect to the cluster that I want to connect to or to the SQL warehouse on Databricks that I want to connect to. And I can retrieve that information from the Databricks comps or the Databricks UI. Here's an example of how we can connect to that, how we can make an odbc connection into Databricks.
And I want to highlight one additional thing here that's that we're excited about. If I if I connect traditionally, so a typical odbc connection from R would rely on the odbc package and then specifically would rely on the odbc function from that package to facilitate a connection. And then that allows me to pass in things like what driver do I want to use and what connection requirements or what connection string details need to be presented or provided to that driver to create the connection to this data source.
In the case of Databricks, I actually need to provide a lot of detail. I need to provide the driver name, the host where the host where my Databricks, where Databricks is being run, the port, the HTTP path, the catalog I want to connect to. I need to define whether I'm using SSL. There's this thrift transport argument. I need to define what auth mechanism I'm using and provide a user ID and password, whether that's a username and password combination or a token and the personal access token that I'm supplying. But I have a lot of details that need to be provided in order to create a connection to Databricks.
Well, because we've done a lot of work inside of Workbench to simplify things like credential management and other components necessary to access Databricks, we're introducing a new function to the odbc package and upcoming release called Databricks. And this function exists to help facilitate connections to Databricks through odbc, whether that driver is the odbc driver that you install from us, or it's the odbc driver and connector that you install and download and install from Databricks themselves.
With this new odbc Databricks function, we can get rid of essentially everything that we saw in the previous connection string and instead just supply the HTTP path that we need. And that's it.
And so to highlight this, if we run this chunk of code right here, we'll see that we have an odbc connection into Databricks. And this odbc connection will give me a similar view into Unity Catalog as what we saw with Sparklyr. So once again, I can expand out and I can look at, okay, here's this new fork schema, here's this table inside of there, here's the different values that are included in that table.
Working with Databricks data in R
Once we've created these connections into the Databricks environment, we can create local references to data tables that might exist in the Databricks environment. So for example, if I run this new fork command here, if I run this line of code here that uses the table function or the TBL function to reference the data inside of Databricks, so notice that I'm referencing demos, new fork, and then the new fork reports, this will create a local reference that points to that data inside of Databricks.
In fact, we can see this because if we run the head function, which would typically return the first six rows of a given dataset, and then we run show query, this will show me how this ends up being translated on the backend in the Spark SQL that gets executed on Databricks and the results come back into my environment. The key here and the benefit of this is that I can keep all of my data in Databricks as long as I need to and work with it there. And then when necessary, I can move whatever summarized version of the data I need back into my RStudio environment.
So if we wanted to, like, for example, right, we had this UFO sightings data. If we wanted to explore this data and gain some insight from it, we could do something like take the new fork data and filter by date time and then calculate by month how many UFO sightings that were from the dataset, and then we could explore what that might look like. We can keep, again, we can keep all the data inside of Databricks to perform the aggregation of this and then finally collect the resulting aggregation into our R session and then work with other tools within R like ggplot, for example, to create some sort of a visualization of whatever it is that we're investigating.
In this case, we're saying how many UFO sightings were there by month, by year in this dataset. And so we can see if we look at this, there's not a lot up until about 2000. Then things started to increase in 2010, maybe 2009. Things really kind of ramped up, but it maybe died down a little bit in the past couple of years. We could also do, you know, we could extend this even further and we could do other analysis, we could do other exploration. But I want to highlight a couple of other things here in the brief time that we have.
Sharing insights with Posit Connect
One is, as a data scientist, this is how I typically spend my time. I work interactively with some sort of a dataset until I find some insight or actionable piece of information from that dataset. At that point, what I want to do is I want to be able to share that insight or that whatever it is that I found with other members of my organization. That can take a number of different forms. I could be an email. I could build a PowerPoint presentation. There's a number of, I can send a Slack message. There's all kinds of different ways that I can distribute information.
But what we found at Posit is that the most effective way to distribute information is to provide something that's reproducible and that's lasting, that users can come back to and reference whenever they need to. And it's difficult to do that with something like an email that can get buried in an inbox or a Slack message that can get buried in a Slack thread.
To help support that idea, we have what I've talked briefly about earlier, a product called Posit Connect. And Posit Connect would allow us to do something like, here's an example where I can take a Quarto document. So this is Quarto, which is a kind of successor to R Markdown, a way of interspersing code with plain text and prose. This is actually, I'm not going to dive too deeply into this, but this is a preview of a new feature that's coming in the next release of Quarto that allows you to create dashboards using the Quarto package.
So I can create this document here that contains R code interspersed with plain text. If I look at this document locally, so I'm just going to go ahead and render this. This will create a dashboard that allows me to view some key summaries of this, in this case, this UFO data.
And once I've created this dashboard locally, right, once I've got all the code working and I have this rendering and looking the way that I want it to look, I can easily publish and distribute this through something like Posit Connect. Once this is distributed on Posit Connect, let's open this up here. So here's an example of this dashboard. I can see over time percentage of reported sightings by different countries. I can also break this down by day of the week or by hour of the day. Looks like most sightings happen in the evening, which is to be expected. Much harder to see bright UFOs in the middle of the day.
I can see a summary of the different types of ways in which these sightings are described, as well as the different ways in which the shape of the UFO has changed over time over here on the right hand side. And then I can also view some additional details about the data here. So this gives me a really nice way to package up some information that might be highly relevant to different members of my organization and then share and distribute it with them.
What that looks like is let me come back in here for just a moment. If I there's a couple of different there's a number of different ways you can deploy and share content on Posit Connect. But if you use the RS Connect package, there's a couple of key advantages, particularly when it comes to working with Databricks supported content like a Quarto dashboard that queries data on the Databricks side. And that is I can define both the Python environment that I want Posit Connect to use.
So one of the advantages of Posit Connect is that it will define whatever environment I have locally, including the R packages and their versions and the Python packages and their versions. And it will send a definition of that environment to Posit Connect. When Posit Connect tries to publish whatever it is that I submitted, in this case, that would be this Quarto document. It will first recreate the environment it needs to run that content and it will do this in isolation. So I do not have to worry about managing a complex network of dependencies on Connect. Instead, every piece of content has its own collection of dependencies.
There's a shared cache, so I'm not duplicating work. But it means that everything can operate in isolation without interference with other content. So if this dashboard uses some version of PySpark that another dashboard uses a different version of PySpark, there's not going to be a conflict when it comes to publishing and sharing on Posit Connect.
The other advantage of using this RS Connect package is that it allows me to pass local environment variables in a very secure way to Posit Connect. So this is how I can pass information about how I'm making the connection to Databricks so that Posit Connect can make that connection itself.
If I look, so here's an example of this document, this dashboard now published and shared on Posit Connect. And now I can come over here and I can say, let's add Rafi to this dashboard. We could have other members of our organization. We could add groups and users. We could be very specific about who has access to this dashboard. We could schedule this dashboard to refresh every week or every day or however often the underlying data is updating. How often do we want to be informed of the latest UFO trends? We can define all of that here inside of Posit Connect and end users can come visit this whenever they want to. It's a specific URL they can log into and view.
Just to highlight one other thing that we could do in this kind of same idea is we could also, if we wanted to provide something that was a little bit more interactive and less static than the dashboard, we could also offer a shiny application. So we could build an interactive web application in R or Python. We could publish that application to Posit Connect and that application would enable users to come and investigate this data on their own and ask questions on their own.
So here's an example of a shiny application using this data that's being hosted on Databricks where we can take a look at, OK, I live in Utah. Here's the collection of UFO sightings that have happened in Utah. Let's filter to not 2010. Let's come. Let's say let's filter from 2012 until now. Search through the records. I can zoom in over here. Take a look. Let's see. I live over kind of in this area. What is what do we have here? OK, here's the report of this of this particular UFO sighting.
Right. And this type of interactivity allows end users not just to ingest the work that's been done for them, but also ask their own questions and get their own answers from the underlying data. And all of the data here is being stored and managed on Databricks through the data governance capabilities that Databricks has to offer. And queries are being submitted and registered against the Databricks computed file.
And this type of interactivity allows end users not just to ingest the work that's been done for them, but also ask their own questions and get their own answers from the underlying data.
Demo recap
OK, so let me let me just take a minute and summarize a little bit about what we've chatted through. Right. What do we have today? Well, Rafi talked to us about VS code and some of the changes that exist, some of the opportunities that exist within VS code and Databricks, the VS code extension for Databricks. We talked about our studio and the Databricks pane and the way in which Posit Workbench can manage authentication and manage credentials for Databricks users. And these are all things that are available either today or in the very near future with the upcoming release of a positive workbench.
We also talked a little bit about sparkly are and all the new changes we introduced there. There's been a lot of work, an incredible amount of work that's gone into the sparkly our package over the past few months to support this new Databricks connect functionality inside of Databricks and to allow and enable our users to seamlessly create these remote connections into the Databricks environment.
Roadmap and the future
Now, for the last couple of minutes before we turn our time to questions, let's talk about the future. And this is what in particular gets me very excited. We've talked a lot about how we can use these two products separately, or these two different platforms separately, we can have positive running somewhere inside of our organization, you can be using our studio desktop and sparkly are to connect to Databricks and execute workloads there.
There's all these different paradigms, but the key for everything we've talked about so far is that the assumption is Databricks is running somewhere and posit stuff is running somewhere else. Maybe it's all in AWS, maybe it's all in Azure, but it's separate in its implementation and in its architecture. And we can see that in this diagram that we chatted about earlier, where we have the Databricks data intelligence platform. And then we have positive team. And there's a number of different ways that they can interact through odbc SDK sparkly are API's, whatever the case might be.
But what we're looking forward to and excited about is in the near future, we look to provide a way to run positive team and positive products directly inside of your Databricks infrastructure and architecture through a platform called Lakehouse applications. Now, Lakehouse applications is something that was announced by Databricks back in the summer of this year. But the idea is that it's a way to support native third party applications running directly on your Databricks infrastructure. In this sort of paradigm, there's no separation of infrastructure and data stays within the realm of Databricks.
When you're querying and moving data from your data warehouse or data Lakehouse into a session that's running on our studio or into an in memory session that's running with VS code. Everything will stay inside of that Databricks security governance layer. Nothing's moving outside of that. And that's something that we've heard frequent feedback from customers on and something that we're excited to look to provide as we look towards the towards the future.
Everything will stay inside of that Databricks security governance layer. Nothing's moving outside of that.
Once again, want to thank everybody for your attendance today and say that we on the positive side, as well as I know that the folks that we work with on the Databricks side are exceptionally excited about the work that's been done and the work that's going to be done in this partnership as we move forward. If there's curiosity on your end, if you're interested in any of the things we've discussed today, we're happy to have conversations about both the current and future state of our partnership to answer your questions to collect your feedback. Now is a great time to have those conversations because we're just getting started in the process of defining what it will look like to run posit products on Databricks. And so if you have strong opinions on how you'd like to see that happen. Now is the time to have those conversations with us.
There's a number of different resources available today to help provide some additional information, along with what we chatted about today. The QR code on the screen now will take you to a GitHub repository that contains the slides, the demo code and everything that we walked through today. And then there's also a series of links here that you can visit from the slides that are available on on that GitHub repository.
With that, I'm going to actually just turn the time over and we'll take some time to answer questions that have come in through the different chat forms. We have a bunch of questions. Should we go from the top? Let's do it. OK. Just catching up on all the all the chat things.
Q&A
OK, how difficult is it to also include other languages like Julia or Mathematica? I'll answer this question from the positive side briefly, and then you might have some thoughts on the Databricks side. I know. So I think probably stems from the introduction that I made to posit how we aim to support open source data science and development, regardless of the language. R and Python have been the primary drivers in that area for a long time. Who knows what the next 10, 15, 50 years are going to bring in that space? But we remain committed to paying attention to what's happening and providing tools. It's possible to use things like Julia inside of some, you know, inside the RStudio IDE today to some extent. You know, you can write some Julia code and things like that. But it's not it's not a first class citizen in that regard.
Same with Mathematica. But a lot of that's just driven by we try to pay attention to what customers are asking for and what sorts of trends we're observing. And so far, that that's been very clear that Python and R have been the two that we we plan to continue to support.
Yeah, I would add that, like on Databricks in particular, Python, SQL, R and Scala are the supported languages, and I don't think there's any plans to add Julia. However, you can always install Databricks is super friendly to open source. So you can always install Julia and run it on Databricks.
Yeah, and I will say there's a number of additional things similar to that on the on the positive side and on the RStudio IDE side specifically. Right. You can, for example, in a Quarto document, you can have bash code chunks, you can have SQL code chunks, you can have Python and R code chunks kind of interspersed throughout the same document. So there's there's some interoperability there. But but Julia and Mathematica aren't ones that are highly prominent right now.
OK, we got another one. Let me see. It's a good app package manager so that every developer can share packages installed on R and Python and don't need to install packages for every developer. So so package manager like package management's an interesting topic because there's there's a number of different pieces to the puzzle. Package manager kind of sits in the middle. So so every package comes from an upstream repository. And then when it's installed, it gets installed into a library. That library can be scoped in a number of different ways.
But that's the gist of it. And package manager allows you to create your own internal upstream repository where you can say, hey, you know, we this is particularly beneficial for things like regulated industries where we might say, hey, we have requirements that prevent us from using these certain packages or packages with this license type. There's some new security features that we're introducing deposit package manager that will allow you to like run or like that will help provide visibility into like vulnerabilities and things like that that may exist in different open source packages. And so package manager allows you to say, hey, like as an organization, here's the packages we define for our users and then users would still install them individually into their own environments. But they would be coming from package manager rather than coming from an unknown repository out there on the open Internet.
In a Databricks notebook, you can directly start a Shiny app. That is true. Will we will we be able to do that with PyShiny as well? I don't know right now. That's something we have to look into. I haven't tried it and it may have a very different implementation. So I'm going to say for now, I don't know if people people try it and it works would be great to know.
Okay, next one. Can the VS code can can VS code be run on a local PC and still connect to Databricks? Yes, totally. If you whether you have VS code running and posit workbench on like a VM in the cloud or workbench running on your laptop. It doesn't really matter. The VS code extension will work wherever you have VS code installed.
Again, the code that you're that I showed you with with Databricks connect when you're basically running spark command spark data frame API against remote database compute. If you want to run more code on Databricks, you can run the entire file on a Databricks cluster, either as a job or as like a one time just quick submission to run it in a batch process. So in that case, you do need to make sure that the libraries that you're that you have installed locally where your code runs locally will also be present in in Databricks.
The easiest way to do that today is to probably go to the database UI and install the packages on the cluster that you want to us that match your local environment. If you want to do it programmatically, the easiest way to do that is to use one of the SDKs to use the cluster API, you can update the packages that are installed on the cluster through the rest API that way.
Okay, I'll take this one. So in the future, will I be able to easily develop, publish and share shiny streamlet or dash apps all natively in Databricks?
That's something that we're looking at very closely. And I think with lakehouse apps, and including with the partnership with with positive, that is something that I think will happen in the near future. I don't have any firm timelines or anything like that. But it's, you know, data apps and hosting data apps is something that's like very, very on our mind.
I think this one's for you, James.
I'm not sure what are. Yeah, I don't I don't really I don't know what's in ours, either. Sorry, I don't either. Maybe we can answer that in the chat or follow up. Let's go to the next question.
In general, any open source package that's not in Databricks runtime, there's a set of open source packages and are in Python that are installed out of the box with Databricks runtime. Anything that's not included in that, and it's an open source, you can install it. If it's your custom package, you can install it. So you can always bring your own libraries in your own code.
All right. Next question. If a Databricks cluster has Python libraries installed that aren't available in the runtime, will those be included in the environment created in Workbench? So sort of going the other way, like if I have libraries in Databricks, and I'm creating a new session in Workbench, am I getting those libraries in Workbench?
So I think the answer for that today is, is no, you would have to get a list of you'd have to programmatically get a list of the packages that are installed in on the cluster on the Databricks cluster. And then once you have that list, you can install them in your Workbench environment. But there's not like an automatic sync of the execution environment to that degree of to that resolution.
There's a lot of questions about that. That's a really it's you're touching upon like an important, all these questions about environment management are very important. And it's something that we're at Databricks, at least we're looking to solve those problems right now.
Running processes on Databricks
Okay, next question. Can you set this up to run the entire our process on Databricks? Or do you have to only run the spark pieces on Databricks and the rest is local. So today, like I said before, the interactive remote connection to Databricks is spark only. However, you can run your entire script or you can run your entire script on Databricks in a batch process. And then that sync that I showed you with the VS code extension.
That actually makes it so that all of your relative paths will work. And that includes for our as well. So that is sort of where we're at with that we are also in an effort to continuously make this experience better and easier. We're trying to figure out ways to have the entire process run on Databricks so that you don't have to have this sort of like split brain between some things run on Databricks and some things run locally. So we're aware of that complexity and we're trying to resolve it.
Feature timelines
Yeah, I can take this one timeline for some of these features and things like that with positive Databricks. So the things that were specific to positive workbench that we showcase. So the authentication management and the Databricks pain and things like that, that those will be made available in the next release of positive workbench, which is expected this month.
The sparkly our stuff is available today. We're continuing to can make some incremental improvements to that package. But but what we've chatted through should be should be available today. You may need the dev version of the package for a little bit of time before it hits Cran, but but it's available.
The odbc stuff is the so for example, that Databricks function from the odbc package that we highlighted, that will be part of the open source odbc package, it just needs to be released to Cran. So if you wanted to use that today, you could, it just needs to be used to use the development version of the package until until the Cran release goes out, and then and then have access to a Databricks driver, either the one that Databricks provides or the one that will begin providing the customers and that driver that we're going to be providing the customers will be available in the next few weeks or so.
Yeah, so let me, I'll answer this one. And just reiterate that yes, the Databricks function from the odbc package will be available via open source, I realized I left off a significant portion of the previous answer, which is when we talk about lake house applications and running positive products on Databricks, and some of the work that's going to happen between now and then.
So one of the things that I haven't mentioned yet is the work that we're doing right now to start looking at how do we support? How do we better support Databricks authentication and credentials and things like that inside of positive connect? And what does that look like? Those are those are questions that we're looking for answers on right now and thinking deeply about. So there's going to be some changes that are introduced there. All of that stuff will happen over the course of next year.
So lake house applications, I think it's something that's will be sometime towards the middle or third quarter of next year. And then there will be some work kind of between now and then to continue continue to improve the way in which positive users and customers and and things like that can benefit from the work that we're doing with Databricks.
Sharing Shiny apps and ODBC capabilities
Okay, I'll take this one. Can I can I share a shiny URL within Databricks without deploying to a shiny server? Yes, you can. So you if you use Databricks notebooks, and you run the shiny app from Databricks notebooks, you will get a URL that you can share with other people. The condition is that they have to be able to sign into Databricks. So if you have single sign on, and people want to sign into Databricks, you can do that.
So if people are able to just like sign in, that's, that's going to be pretty nice experience, they'll be able to access the shiny app directly. Otherwise, you know, that's not going to work. But fundamentally, yes, that is that is possible. Okay, let's go to the next one.
I can answer this one. I can take this one. Oh, yeah, go for it. All right. Next one's yours. Next. Next tie goes to you. All right. Does Databricks odbc driver handle things like spark ml library? Or is it just more simple data pulls data access. So odbc is SQL only. It's essentially only going to use the spark API, the data frame API and pass SQL down to the spark cluster. There's no support through odbc for Spark.
ml, or for streaming or for graph or anything like that. Okay, I think that's our last one for today. Yeah, go ahead. Yeah.
No, I was just gonna say thank you, everyone for coming. We're again, just reiterate, very excited about where things are today and really very excited about where things are going. If you have questions, thoughts, comments, feedback, feel free to reach out. There's lots of different ways you can get in contact with us. Feel free to take these things out for a test drive and let us know what you find. And yeah, stay tuned for more. We're excited and thankful for all of you for joining us today.
We're again, just reiterate, very excited about where things are today and really very excited about where things are going.
Absolutely. Thanks for all the engagement for your time and we look forward to seeing you next time.
