Azure Synapse Analytics in 2022: Retrospective
Retrospective
Another year has rolled by (where does the time go…) and another year in the life of Azure Synapse Analytics. I’ve mostly spent time blogging and talking about Synapse Serverless SQL Pools, how Power BI can plug into and take advantage of Serverless. I really do love Serverless SQL Pools, easy to setup and work with, but limiting in terms of what is provides – which can be a good thing, nice and focused.
Ultimately has Microsoft done enough to keep progressing Synapse Analytics? I’m not that interested in flashy announcements, but I do expect a steady stream of improvements. Other vendors are pushing in terms of advancing their platforms, Microsoft needs to do and keep doing the same thing – push, innovate, and improve. We can do out bit and keep Microsoft on their toes by demanding more.
From my perspective I really would have liked to see more from the Dedicated SQL Pools service, where was auto-pause/resume, auto-scale? Serverless SQL Pools had some new features which were nice, but Microsoft have a service there they can really get behind so more needs to be done in that area to really compete with other vendors in the lakehouse space.
Did you find out about Synapse Analytics, did you migrate to or away from Synapse? Are you using it and are satisfied/dissatisfied? Do you love it/hate it? The comments are open so feel free to discuss. This is not a blog post designed to criticise anyone or the Synapse team, they work hard and I support and appreciate that. Let’s make sure any criticism is constructive and also send praise – an imbalance on either side isn’t very useful.
Convergence of Synapse and Power BI
Back in April we had the release of the new Microsoft Certified: Azure Enterprise Data Analyst Associate certification backed by the DP-500 exam, which covers both Power BI and Synapse Analytics. The certification occupies the space between the PL-300: Microsoft Power BI Data Analyst and the DP-203: Data Engineering on Microsoft Azure. I’ve seen a lot of people in the Analyst space really excited about this and be excited they can learn more about Synapse.
We had Adam and Patrick from Guy In A Cube making Synapse videos, who saw that coming eh! It makes sense as Power BI works very nicely with Synapse and there are a whole bunch of features in each service that as a whole solution can provide very fast data access and processing.
In 2022 and into 2023 I see lots of sessions championing Power BI and Synapse Analytics together, there are several sessions at SQLBits 2023 focused on this plus a training day with Guy In A Cube.
Marketing & Promotion
This has always been a bit of a grey area for me, can marketing and promoting a data warehouse/big data service work via social media channels and not just corporate channels? That person who saw a tweet from the Synapse Twitter account might end up watching a video and buying into the service…who am I to argue with that?! Was 2022 the year that Synapse spread via viral marketing?
We saw the official Synapse Analytics YouTube making more videos including tutorials, guests, diving into Synapse services and features (yours truly guested as well, who let me on eh!). Ryan Majidimehr and the team are doing a great job to keep us all updated with the latest info and provide a steady stream of useful videos.
Synapse Espresso started up too, with around 15 videos covering Dedicated and Serverless SQL Pools, Stijn and Filip have regular guests on and dive into a particular service or feature in each video. I honestly look forward to these videos as I always end up learning something new (calling out the Serverless SQL Pools statistics video as a personal favourite!).
We also saw the launch of the Synapse Influencer Program (link) in May. Yes, you heard that right…an influencer program….for Synapse. Well, people got right on board and started posting content, spreading the word, and it genuinely feels like a community.
Shout out to Pawel Potasinski (T), Senior PM on the Azure Synapse Analytics team, who continues to be a positive driving force between the data community, the Synapse team, and Microsoft – a real credit to Microsoft.
Data Toboggan – Synapse Analytics Conference
Richard Munn, Victoria Holt, Mark Pryce-Maher, and myself carried on Data Toboggan into 2023 with 3 Azure Synapse Analytics focussed events plus the monthly meetup group run by Richard. We built a new website here with all the latest event news too. All the available sessions are available on the YouTube channel here. We had the wonderful Kim Manis (Partner Director of Product Management, Azure Synapse Analytics & Power BI at Microsoft) provide us with great opening and closing keynotes for the events too, thanks Kim!
We have our Winter Edition happening on Saturday January 28th, you can register here.
Monthly Updates
Well this was a welcome addition (well it actually started sometime in 2021) but IMHO a data warehouse /big data analytics platform probably isn’t agile enough to have a whole bunch of new features every month…and to be honest I wasn’t expecting huge changes month by month. Also, do you want masses of changes every month on a system that serves as the backbone of a data warehouse/analytics system?
I’m not much of a Spark developer so I tend to ignore the Spark news. This is something I’m remedying for 2023 as I look to use Spark for more data workloads (this includes Databricks usage too). Here are the SQL Pools announcements made in 2022 through the monthly updates.
Dedicated SQL Pools
- COPY schema discovery for complex data ingestion (link)
- Column Level Encryption for Azure Synapse dedicated SQL Pools [Generally Available] (link)
- Cross-subscription restore for Azure Synapse SQL [Generally Available] (link)
- Recover SQL pool from dropped server or workspace (link)
- Data Warehouse Migration guide for Dedicated SQL Pools in Azure Synapse Analytics (link)
- Synapse Success by Design (link)
- Reduce cost with batch mode in Synapse link for SQL (link)
- Multi-column Distribution is available for public preview in dedicated SQL pools (link)
- Distribution Advisor is available for public preview in dedicated SQL pools (link)
- Accelerate your Oracle data warehouse migration to dedicated SQL pools in Azure Synapse Analytics (link)
- MERGE T-SQL is now Generally Available (link)
Serverless SQL Pools
- HASHBYTES easily generates hashes in Serverless SQL (link)
- More consistent query execution times for Serverless SQL Pools (link)
- The OPENJSON function makes it easy to get array element indexes (link)
- Better performance for CETAS and subsequent SELECTs (link)
- Database Designer industry templates revamped exploration experience (link)
- Clone Lake database (link)
- Use wildcards in Database Designer tables to specify custom folder hierarchies (link)
- Specify character column lengths… Not anymore! (link)
- Result set size limit increase (link)
- Add SQL objects and users in Lake databases (link)
- Spark Delta Lake tables are available in serverless SQL pool for public preview (link)
- Auto-statistics for OPENROWSET in CSV datasets (link)
Discussion
Of course there were social media discussion, debates, hot takes…it wouldn’t be a community without it! I appreciate it all and can take the odd punch to the ribs or two ;). If I were to single out 1 person in the community that keeps us on our toes that would be Mimoune Djouallah (T)…. oh Mim, you keep us all real and I love the debates. Keep it coming!
2023
And into 2023 we go, there’s plenty to crack on with in terms of Synapse. Look out for my Synapse Link videos on YouTube (btw Datahai is my company), there will be plenty more DP500 Enterprise Analyst focus, and also some DP203 Data Engineering focus too. There will also be a series of posts in building a lakehouse solution using Serverless SQL Pools. I am in the process of learning for the Databricks Data Engineer certifications too, it’s not all Synapse you know 🙂
Happy New Year to you and I hope you have a great one!
Thanks for all of the good links you have added in your post. I now have more places to go read about Synapse. I liked your year in review as you raised some good points and reminded me of how different things were last January.