Fabric Networking: Managed Private Endpoints with Setup and Considerations

Blog Contents

Managed Private Endpoints in Public Preview

Security fans rejoice (and companies that don’t appreciate leaving their data stores open to public access…)! Microsoft Fabric now has Managed Private Endpoints (MPE for the remainder of this blog) in public preview. For those not familiar with MPE, this feature allows a service such as Azure Synapse Analytics, Power BI, and now Fabric, to connect to another service that has been disabled from any public network access and/or hidden behind a firewall. E.G. you can create a storage account and disable it from public access but services will be able to connect to it via MPE. The beauty of MPE is there’s very little configuration to enable and connect, and for someone not that network savvy like me, it works very well to secure services.

Announcement here: Introducing Managed Private Endpoints for Microsoft Fabric in Public Preview | Microsoft Fabric Blog | Microsoft Fabric
Documentation here: Overview of managed private endpoints for Microsoft Fabric – Microsoft Fabric | Microsoft Learn

The ability to connect to Azure services securely is probably one of the most asked questions on forums (including over at the Fabric Community site) so I’m very happy the MPEs are now available. I’ve blogged about setting up connectivity from Power BI to MPE enabled services before.

You can create MPEs in Fabric Workspaces by using the Network Security tab and specifying the name you wish to give the MPE, the Resource ID of the service you wish to connect to, and the resource type. We’ll go through this in the steps below.

UPDATE 1st March 2024: There is new functionality to allow Lakehouse Shortcuts to access secure data stores: read more here. In this blog we’ll cover using Notebooks and Spark to connect to the secured storage account. I previously included Shortcuts in this blog but I have moved into a separate blog.

Considerations

Always read the considerations and limitations of new features and services, some limitations do surprise me…

Here’s the biggest catch, only F64 and above, plus Fabric trial capacities are supported. So no luck for anyone trying to implement this under F64. That’s a shame. I’ve reached out to MS to clarify any pricing details, but I’m assuming the reason an F64 is required is that normally a MPE costs ~£5 a month for 1 MPE (pricing here).
An annoying limitation is that the Starter Pool in Spark is not available when a Workspace has a MPE configured, so expect those session start times to increase to 2-3 minutes.
Why are Lakehouse table maintenance operations not supported in workspaces with MPE enabled? No idea, but they’re not…
Read up on the other considerations here

Prerequisites

A Fabric trial capacity or Fabric F64 SKU (or above). Power BI P SKUs are not supported currently.
A Workspace allocated to the Fabric/Power BI capacity

Creating Managed Private Endpoints

In this example we’ll create an MPE to an Azure Storage account (configured as a Data Lake account) which has been disabled from public access at https://storqasynapsedata.dfs.core.windows.net/

Login to Fabric/Power BI and browse to a Workspace assigned to a trial or at least an F64 capacity
Go to Workspace Settings and click the Network Security tab
You can then click the Create button

Go to the Azure service that the MPE needs to connect to e.g. storage account in the Azure portal
Under the Settings section, click Endpoints
Look for the Storage account resource ID and copy this into the Resource Identifier field

The MPE should now be registering and after ~1 minute you should see Succeeded in the Activation status. In the image below I have created several MPEs.

You’ll need to go back to the resource (storage account) in the Azure portal, go to the Security + networking area, click Networking and select the Private Endpoint Connections tab. Select the newly created MPE and click Approve.

The MPE should now show as Approved, this can take a few minutes so hit the Refresh button until to see the status updated.

Connect to Secured Storage using Notebooks

Within Notebooks in the MPE enabled Workspace, I was able to use the full ABFSS path to access the secured storage account. I noted that if I tried to access the remote storage from another workspace using the actual abfss path I received an access denied error.

df = spark.read.parquet("abfss://data@dhstorqasynapsedata.dfs.core.windows.net/read/*.parquet")
display(df)

However this all comes with a caveat as the quick start Starter Pools are not available, I received a message in the Spark Settings area of the Workspace:

“Starter pools aren’t available for Fabric workspaces using Private Endpoints. When Private Endpoints are created, new Spark sessions will take 2 to 3 minutes to start instead of 5 seconds.”

And yes, the session did indeed take around 3 minutes to start..

Conclusion

I’m relieved that Managed Private Endpoints have been released into the Fabric eco-system, it’s essential to keep all your data resources as secure as possible and this is again another way to lock down your data estate. I’ll be keeping an eye out on the functionality going forward.

As always, get in contact here to discuss anything in this blog.

Tags: Fabric, Managed Private Endpoints, Security