T-SQL Tuesday #97 – Learning Goals for 2018

T-SQL-Tuesday-LogoThis month’s T-SQL Tuesday is brought to us by Mala Mahadevan.  Mala has challenged us to set some goals for our own learning in 2018. This could be learning to improve our social skills, technical skills, or perhaps some other area of our lives.  The challenge is to be intentional about our learning by answering three questions:

  1. What do you want to learn? (specific skills and talents)
  2. How and when do you want to learn? (methods of learning and timeline on learning)
  3. How do you plan to improve on what you learned? (Putting it to use at work/blogging/speaking)

I know that I’ve wound down the year with some time to learn new skills, sometimes out of necessity, sometimes because the end of the year has allowed a little more freedom to pursue that time to learn. One year I taught myself enough Python to script out some tasks that were giving me huge amounts of trouble at work. I figured out how to talk to SQL Server, manipulate files, call out to command line utilities, and enable some easier re-use of the code so it could be parameterized for similar uses with different customers. This was back when PowerShell was still just hinted at by MS and their only options were VBScript, Batch files, or some 3rd party option.  Had this come a couple of years later, I’d likely have tried to do the same in PowerShell.

What Do I Want to Learn?

In 2018, I can see a couple of areas that I’d like to grow in.

  • Biml – I see quite a bit of work coming soon that will have me working on similar SSIS packages that will require tweaks and adjustments. I really don’t want to repeat some of the fun of “adjust a UNION ALL” transform and click through the package to adjust the rest of it. If I can get enough Biml under my belt to even start to automate this or reduce my work, it will be a win. We’re looking at doing ETL from one source to many databases as well to support Power BI. I can see Biml being a powerful tool to help with that.
  • Power BI – What can I say? Data Visualization is big and MS has been investing heavily in Power BI. It’s what our customers want and what we’re planning to use to enable them to see all sorts of interesting data. I need to keep up with what MS is doing here and be able to support loading and querying the data so we can present it to our customers. I still plan to leave the “make pretty” work to those who are more talented than I in that area.
  • PowerShell – With more of my work going to Azure SQL, I really need to ramp up my PS skills. I’ve primarily worked as a DB Developer – TSQL, SSIS, SSAS, some SSRS, and design. PowerShell hasn’t been a high priority there because, apart from deploying those things, I don’t really have a need for it day to day.  With Azure SQL in the mix, if I want to query all of our databases, I no longer have the ability to use something like “sp_msforeachdb” or run a cross-database query. I need to connect to each DB separately, run my code, then connect to the next.  Knowing more PowerShell will help me quite a bit to handle those repeated tasks.

How and when do I want to learn?

Now for more details. This is where things get harder to plan, but where a plan is most needed.

Biml – my main starting point here will be to get through the Biml Book that was just released by APress.  I’ve already started reading through this and have worked with Biml before, though not to the extent of others. I’m aware of its capabilities and know that it would help me greatly. I will need to ramp up some C# skills to be able to make more effective use of my code, but even without that I can help make packages that are easier to tweak.  My plan here is to get through about one chapter per week.  I’ll then take that knowledge and rewrite some of our existing SSIS packages into BIML code.  Any new ETL tasks that come up I’ll try to generate them in Biml so I can materialize packages that are easier to change as needed.
Goal: Able to handle simple-medium level complexity tasks in Biml by June 2018

Power BI – My biggest gains here will be through better understanding of DAX and the Tabular model. I’ve followed the PowerPivot Pro team, regularly read the writings of Russo and Ferrari, follow Adam Saxton’s “Guy In A Cube” videos, and try to keep up with the PowerBI User Groups.  However, I’ll need to take some time to really go through a good Power BI book or course. I plan to use Pluralsight, EdX, and any decent books that I can find to get me beyond the beginner stage for DAX and Tabular data.  This is an ongoing area for me. My goal is to be able to design optimized data structures and answer some more complex questions, especially when it comes to comparing data over similar periods
Goal: Complete the EdX PowerBI course by March 2018. Complete one book on Tabular/DAX by end of 2018.

PowerShell – I’ve gone down this road before. In the early PS days, almost all of the books/articles dealt with administration of Windows domains/services with very little about SQL Server. Even the SQLPS module got little love and community alternatives sprang up. That’s a little different now. My first priority will be connecting to Azure SQL Databases to run queries – especially ones that need to run against a list of databases. After that, I’ll move on to things like deploying and building SSDT Projects, capturing the scripts/builds for later replay and such. There are more tasks for DB Developers now with automated releases, builds, CI/CD, and a plethora of tools that weren’t around 10 years ago.  I can take advantage of books and the SQL Community’s contributions to learn more about PS.
Goal: Complete at least one PS book by the end of 2018. Write a couple of scripts or modules to run SQL against a list of databases/instances/servers in Azure SQL by June 2018 at the latest.

How Do I Plan To Use These Skills?

Most of these will be applied directly to my current job. BIML to build SSIS packages and make them easier to generate – ideally moving towards a metadata-driven framework that I’ve seen others use.  PowerBI will be used to visualize data about all sorts of things – SSIS, performance, customer data/metrics.  I’ll use PowerShell to help me run SQL scripts against multiple databases. I’ve had a need for this for a while and it’s just going to grow when using Azure SQL. I’ll also use it to help me build and deploy database changes to those databases.


TSQL Tuesday #96: Folks Who Have Made a Difference

T-SQL Tuesday Logo

This month’s T-SQL Tuesday, hosted by ewald, touches on the people who have made a positive difference in our journeys in the SQL Server world. Please pardon if you’re reading this and feel left out – that’s not my intention. There are so many people who’ve made a difference for me along the way that it’s hard to list them all.

My first shout-out goes to the folk I worked with way back when I was just starting out. I worked for a company just coming out of start-up mode with SQL Server 4.21a.  The first time I heard of “sequel” server had me scratching my head until I saw the posters around for “SQL” Server. I was given an opportunity to learn, mostly through BOL, SQL Server and become an MCSE through much studying and some horrible exams. I discovered that I kind of liked this database thing and learned more, got better at my TSQL, and learned where to ask questions from the people who were really involved in the product.

I owe a huge debt to the folk who regularly participated in the old “microsoft.sqlserver” USENET groups. I could search through those posts for people with similar problems to what I was trying to solve, or just interesting questions. I would find out pretty quickly about major bugs (anyone remember SQL 6.5 SP2 and SP4?).  Regulars there such as Joe Celko, Steve Jones, Andy Warren, Andy Leonard, and Jamie Thomson would all respond pretty quickly and often provide additional food for thought.  I could follow along with people wanting to argue with Mr. Celko about proper ANSI SQL – usually good for some amusement. I could learn some interesting tips for making DTS jump through the hoops necessary to move data around.  One thing I learned from those USENET groups is they were vastly different from other tech groups at the time. If I asked a question in those forums, I got a real response – not just a terse “RTM and go away” that I saw so often in other forums. I knew that the SQL Community was much more welcoming than other groups at the time and that was quite attractive.

I have to give a shout-out to co-workers along the way who challenged me and let me bounce ideas off of them, as well as for sharing a laugh when the situation didn’t really call for laughing. Jeff Rush, Mark Hill, Nick Floyd, and many of the people from Fellowship Tech, several along the way from First City/Drive/Santander, and many at my current gig at Healthwise.  I appreciate the desire to follow through, think of new ways to solve problems, and do all sorts of cool things along the way to increase our efficiency by spending less time on “keeping the lights on”. As an admitted “lazy DBA”, I can definitely appreciate time spent doing meaningful work instead of busywork.

The people in the SQL Community – many of whom I have not met face to face, but chat with in the Slack forums.  There are just too many to mention here, but I appreciate the regulars who share a joke, a neat way of doing things, ask questions that make me think, or answer questions I have because they’ve been there already.  If you’re in the Slack community – you’re likely in this group. Thank you. I appreciate you all taking part not just in general chit-chat, which is fun, but in helping everyone grow in their skills.

The folk in the PASS Prayers group. As a Christian, it’s great to have a group of people who will pray for each other. Yes, we have that in our local communities, but it’s great to have a group of people who share the same types of career and can join together with a bit more understanding about our day-to-day struggles. It’s also great to meet up with them when I actually get to attend the Summit.  Thanks to Mike Walsh and others for getting us all together.

I know there are groups that I’m leaving out and that’s because there are so many people who’ve been helpful in this journey. Thanks for taking the time to walk with me or guide me. I hope that I can provide that same assistance to others along the way.  And a hat-tip to Ewald Cress for a timely TSQL Tuesday Topic.

Handling Late-Arriving Members in SSIS

I was chatting with some other SQL Community members recently and the topic of handling late-arriving members for a warehouse load came up. I figured it was worth taking a couple of minutes to share the general concept I’ve been using to handle late-arriving members in SSIS. The steps are relatively straightforward if you’ve done much SSIS, but sometimes the little things can trip you up. The example below is based loosely on the WideWorldImporters database samples from MS.

The general steps are

  1. Set up your source query.
  2. Pass the data through a Lookup for your Dimension with the missing results routed to a “No Match” output.
  3. Insert those “No Match” rows into your Dimension using a SQL task – checking to make sure that this particular row hasn’t already been inserted (this is important).
  4. Do another lookup using a “Partial Cache” to catch these newly-inserted members.
  5. Use a UNION ALL transform to bring the existing and late-arriving members together.

Inferred Member Data Flow

Sample code for the Inferred Insert:


SELECT @CustomerID = ?


               FROM   WideWorldImportsDW.Dimension.Customer

               WHERE  [WWI Customer ID] = @CustomerID)

  INSERT WideWorldImportersDW.Dimension.Customer

         ([WWI Customer ID],


          [Bill To Customer],


          [Buying Group],

          [Primary Contact],

          [Postal Code],

          [Valid From],

          [Valid To],

          [Lineage Key])

  VALUES ( @CustomerID,— WWI Customer ID – int


           + Cast(@CustomerID AS NVARCHAR(10)),— Customer – nvarchar(100)

           N”,— Bill To Customer – nvarchar(100)

           N”,— Category – nvarchar(50)

           N”,— Buying Group – nvarchar(50)

           N”,— Primary Contact – nvarchar(50)

           N”,— Postal Code – nvarchar(10)

           Sysdatetime(),— Valid From – datetime2(7)

           ‘9999-12-31 23:59:59.9999999’,— Valid To – datetime2(7)

           2 — Lineage Key – int


ER Diagram Software Options for SQL Server

I’ve seen this come up several times recently in the SQL Server Community. There are diagrams built in to SSMS, but they’re a bit limited in their usefulness. You can’t easily break them down into sub-models or tweak them without affecting the underlying database. There are 3rd party diagramming options for SQL Server, some free and some paid.

Simple-Talk ran an article calling for people to submit their favorite ERD software.

Database Answers has a page dedicated to modelling tools. I’m not sure when the page was last updated, but as far as I know the author is still maintaining the page as it points to some more recent offerings.

Database Answers also pointed to a Wikipedia page on Data Modelling Tools.

Robert Sheldon has a great article on Simple-Talk reviewing Online ER Software offerings. Vertabelo gets his top review, but others are mentioned.


I’ve used ER/Studio for years and generally prefer it to ErWin, though that is probably moot at this point seeing that Embarcadero bought ErWin. It’s a pretty expensive piece of software, though. Not everyone can afford the licenses needed to run it. I hope the pointers to some other lists will help people find the right tool for their needs.

SSRS, Data Sources, Stored Credentials, and Service Accounts

I was helping with the setup of a new SSRS instance on our servers recently and was puzzled when we tried to configure the data source. I wanted to set up a shared data source with a service account that would have the appropriate permissions. All reports were going to be automated – no direct user access. Security in that case wasn’t a huge concern. I pulled up the data source in the Report Manager, entered the credentials, and tested the connection, only to be given a “Login Failed” message.

I checked my username and password – both correct. I tried my credentials – they worked. I tried the service account again – failure.  I verified the service account’s permissions on the target server – correct. Finally, I dug into the Event Logs and noticed that the service account was requesting an interactive login when it failed.  I played around with different groups and found that it would work if in the Administrators group. Not wanting to leave it there, I removed it from the group and did some more digging.

I found this article on MSDN that indicated the accounts used for stored credentials needed both “Allow Log On Locally” and “Log On As a Batch Job” permissions.

I launched “secpol.msc” from a Run dialog and checked the Local Policies –> User Rights Assignments. This showed me that the only group/user allowed to log on locally was the Administrators group. I changed that to add this one service account.  I checked the groups allowed to log on as a batch and verified that this account was included in one of the allowed AD groups.  After setting this permission in the security policy, I tried again – this time to a “Success” message.

SSIS Catalog (SSISDB) Cleanup

I really appreciate what MS has done w/ the SSIS Catalog. We have built-in logging at a level that wasn’t possible in prior releases, but that comes at a cost. The default retention is pretty high (365 days) and MS chose to handle cleanup using cascading deletes. This sort of coding makes life easier for the developers, but almost always performs poorly. That’s especially the case when you have 1 parent row with 100’s of thousands or child rows related to it.  The problem is compounded because the first time people realize they need to lower the retention limit is about the same time that the database is starting to fill up or has filled up. At that point, it’s too late to lower the retention by a large number because the corresponding delete will still cause issues.

I knew someone had written some code for this and a quick search on StackOverflow found this question and the answer from Bill Fellows ( b | t ) :

I’m including the code here for a quick reference, though I recommend reading his answer, up-voting it, and checking out his other resources. The only change I’d consider making is perhaps clearing out the child tables in batches of 1000-5000 rows at a time instead of all rows. I found that I had to work in increments of no more than 10 days at a time to delete history. Any more than that and the time to execute increased quite a bit. In this case, look for your max days currently retained, and drop the @DaysRetention variable to that less a couple of days to see what is removed. Keep dropping that until you reach the point at which you want to stop, set your SSIS Catalog retention days to that value, and you should be good from that point on.

It’s probably worth noting that in SQL Server 2012, MS didn’t include proper indexing by default. You may need to add some indexes in order to have the delete perform in a timely manner and avoid excessive table scans. The index issue was addressed in SQL Server 2014 and higher (and I think in SQL Server 2012 SP2 as well).

Troubleshooting Database Mail on SQL Server 2016

tl/dr – Install the .NET Framework 3.5 feature on your server

I was setting up a new SQL Server 2016 machine today and going through the normal configurations – setting permissions, setting backup/data locations, configuring database mail for alerts, and such.  I had no issues until I started testing database mail and getting … nothing.  No emails received, nothing in the logs, a vague hint that there were 3 pending entries in an external queue, but overall – nothing. I hit the MS Troubleshooting page for DB Mail and verified my configuration was correct. I tested connectivity to the SMTP server, verified the firewall settings, and even TELNET’d to the box.

I finally pulled up Sean McCown’s Troubleshooting Guide for DB Mail and started stepping through it.  I got to the “run the DatabaseMail.exe app” step and was then greeted by a big message telling me that the .NET Framework v3.5 couldn’t be installed.  That rang several bells and I remember reading before that SQL Server 2016 still required .NET 3.5 in order to use DB Mail. I added the feature and kicked off the app again just to see if that would work. No errors and all queued mails were received shortly afterwards.

Quick thanks to Sean for the very thorough guide, complete with examples, and for pointing me in the right direction. I definitely will remember this for the next 2016 server I need to configure and will make sure our Ops team adds the framework as part of their server config scripts for SQL Server.

Long-Running Queries and Extended Events

This has been something I’ve wanted to investigate for a while now. I’ve know you could use Profiler and set up server-side traces to capture long-running events, but was curious how to do the same with Extended Events. I then came across this post from Pinal Dave ( b | t ) that pointed me in the right direction. I followed along with the guidelines he was suggesting and was having trouble finding the “Duration” filter. Turns out I had a bit too much selected in my filtering options or perhaps the Wizard was giving me fits seeing it, but I eventually selected just the Batch Completed or RPC Completed events to see and set the Duration filter. The one change that I’d make from Dave’s script is to set the duration to 500,000 because Duration in SQL 2012 is in microseconds, not milliseconds. I also want queries longer than 5 seconds to start.

I’m including my script below for tracking queries taking longer than 5 seconds.


SQL Server 2016, Database Mail and .Net 3.5 Framework

I came across this post in the SQL Server Slack chats and wanted to make sure that this information was shared so others don’t fall into this trap. Not having .NET 3.5 installed w/ SQL 2016 means no DB Mail and worse, no errors or alerts that it’s not working. I hope that MS will address this a bit more if they keep saying we don’t need .NET 3.5 for SQL 2016. In the meantime, if you want DB Mail, you’ll need the framework, too.


There were so many cheers when Microsoft announced that the .Net 3.5 Framework was no longer a pre-requisite for installing SQL Server starting with SQL Server 2016. Folks were so excited, until they started testing certain pieces of the product, specifically Database Mail. That’s right, if you want to use Database Mail in SQL Server 2016 you have to install the .Net 3.5 Framework.

If you are installing SQL Server 2016 on Windows Server 2012 R2, this might be a problem for two reasons. First, the .Net 3.5 Framework is no longer included by default when spinning up a new install. Second, you will get no errors when testing Database Mail in SQL Server 2016. Mail will be queued, but will not be sent and there will be no errors in the Database Mail log, the SQL Agent log or even the SQL Server error log.

So if you’ve gone…

View original post 50 more words

Getting All Permissions for Server

We recently had to work out a way to fetch and display all effective permissions granted to our Production SQL Server. I had been required to get all permissions at the DB level before, but even then that tended to be more granular and only showed to what roles you belonged and what explicit permissions you’d been granted. We wanted to run something through a loop of all users and get that information into a report. It turns out this was easier than I’d expected, especially after my first attempts to dig through the system tables.

We used a script something like the following:

EXECUTE as login = ‘DomainName\Login’;
select ‘DomainName\Login’ AS Login, * from fn_my_permissions (NULL, ‘SERVER’);

With a little work this can be used within PowerShell or SQL to loop through all logins and capture the information in a table or report.


The function could also be used to loop through all databases by using the database and then substituting Database for Server above:

USE AdventureWorks2012;
EXECUTE as login = ‘DomainName\Login’;
SELECT ‘DomainName\Login’, * FROM fn_my_permissions (NULL, ‘DATABASE’);

For an interesting take on revamping sp_msforeachdb, check out this script from Edward Pollack (t). It might help you if you need to loop through all of your databases to run something like this for your auditors.