ER Diagram Software Options for SQL Server

I’ve seen this come up several times recently in the SQL Server Community. There are diagrams built in to SSMS, but they’re a bit limited in their usefulness. You can’t easily break them down into sub-models or tweak them without affecting the underlying database. There are 3rd party diagramming options for SQL Server, some free and some paid.

Simple-Talk ran an article calling for people to submit their favorite ERD software.

Database Answers has a page dedicated to modelling tools. I’m not sure when the page was last updated, but as far as I know the author is still maintaining the page as it points to some more recent offerings.

Database Answers also pointed to a Wikipedia page on Data Modelling Tools.

Robert Sheldon has a great article on Simple-Talk reviewing Online ER Software offerings. Vertabelo gets his top review, but others are mentioned.

 

I’ve used ER/Studio for years and generally prefer it to ErWin, though that is probably moot at this point seeing that Embarcadero bought ErWin. It’s a pretty expensive piece of software, though. Not everyone can afford the licenses needed to run it. I hope the pointers to some other lists will help people find the right tool for their needs.

SSRS, Data Sources, Stored Credentials, and Service Accounts

I was helping with the setup of a new SSRS instance on our servers recently and was puzzled when we tried to configure the data source. I wanted to set up a shared data source with a service account that would have the appropriate permissions. All reports were going to be automated – no direct user access. Security in that case wasn’t a huge concern. I pulled up the data source in the Report Manager, entered the credentials, and tested the connection, only to be given a “Login Failed” message.

I checked my username and password – both correct. I tried my credentials – they worked. I tried the service account again – failure.  I verified the service account’s permissions on the target server – correct. Finally, I dug into the Event Logs and noticed that the service account was requesting an interactive login when it failed.  I played around with different groups and found that it would work if in the Administrators group. Not wanting to leave it there, I removed it from the group and did some more digging.

I found this article on MSDN that indicated the accounts used for stored credentials needed both “Allow Log On Locally” and “Log On As a Batch Job” permissions.

I launched “secpol.msc” from a Run dialog and checked the Local Policies –> User Rights Assignments. This showed me that the only group/user allowed to log on locally was the Administrators group. I changed that to add this one service account.  I checked the groups allowed to log on as a batch and verified that this account was included in one of the allowed AD groups.  After setting this permission in the security policy, I tried again – this time to a “Success” message.

SSIS Catalog (SSISDB) Cleanup

I really appreciate what MS has done w/ the SSIS Catalog. We have built-in logging at a level that wasn’t possible in prior releases, but that comes at a cost. The default retention is pretty high (365 days) and MS chose to handle cleanup using cascading deletes. This sort of coding makes life easier for the developers, but almost always performs poorly. That’s especially the case when you have 1 parent row with 100’s of thousands or child rows related to it.  The problem is compounded because the first time people realize they need to lower the retention limit is about the same time that the database is starting to fill up or has filled up. At that point, it’s too late to lower the retention by a large number because the corresponding delete will still cause issues.

I knew someone had written some code for this and a quick search on StackOverflow found this question and the answer from Bill Fellows ( b | t ) :
http://stackoverflow.com/questions/21781351/how-can-i-clean-up-the-ssisdb

I’m including the code here for a quick reference, though I recommend reading his answer, up-voting it, and checking out his other resources. The only change I’d consider making is perhaps clearing out the child tables in batches of 1000-5000 rows at a time instead of all rows. I found that I had to work in increments of no more than 10 days at a time to delete history. Any more than that and the time to execute increased quite a bit. In this case, look for your max days currently retained, and drop the @DaysRetention variable to that less a couple of days to see what is removed. Keep dropping that until you reach the point at which you want to stop, set your SSIS Catalog retention days to that value, and you should be good from that point on.

It’s probably worth noting that in SQL Server 2012, MS didn’t include proper indexing by default. You may need to add some indexes in order to have the delete perform in a timely manner and avoid excessive table scans. The index issue was addressed in SQL Server 2014 and higher (and I think in SQL Server 2012 SP2 as well).

Troubleshooting Database Mail on SQL Server 2016

tl/dr – Install the .NET Framework 3.5 feature on your server

I was setting up a new SQL Server 2016 machine today and going through the normal configurations – setting permissions, setting backup/data locations, configuring database mail for alerts, and such.  I had no issues until I started testing database mail and getting … nothing.  No emails received, nothing in the logs, a vague hint that there were 3 pending entries in an external queue, but overall – nothing. I hit the MS Troubleshooting page for DB Mail and verified my configuration was correct. I tested connectivity to the SMTP server, verified the firewall settings, and even TELNET’d to the box.

I finally pulled up Sean McCown’s Troubleshooting Guide for DB Mail and started stepping through it.  I got to the “run the DatabaseMail.exe app” step and was then greeted by a big message telling me that the .NET Framework v3.5 couldn’t be installed.  That rang several bells and I remember reading before that SQL Server 2016 still required .NET 3.5 in order to use DB Mail. I added the feature and kicked off the app again just to see if that would work. No errors and all queued mails were received shortly afterwards.

Quick thanks to Sean for the very thorough guide, complete with examples, and for pointing me in the right direction. I definitely will remember this for the next 2016 server I need to configure and will make sure our Ops team adds the framework as part of their server config scripts for SQL Server.

Long-Running Queries and Extended Events

This has been something I’ve wanted to investigate for a while now. I’ve know you could use Profiler and set up server-side traces to capture long-running events, but was curious how to do the same with Extended Events. I then came across this post from Pinal Dave ( b | t ) that pointed me in the right direction. I followed along with the guidelines he was suggesting and was having trouble finding the “Duration” filter. Turns out I had a bit too much selected in my filtering options or perhaps the Wizard was giving me fits seeing it, but I eventually selected just the Batch Completed or RPC Completed events to see and set the Duration filter. The one change that I’d make from Dave’s script is to set the duration to 500,000 because Duration in SQL 2012 is in microseconds, not milliseconds. I also want queries longer than 5 seconds to start.

I’m including my script below for tracking queries taking longer than 5 seconds.

 

SQL Server 2016, Database Mail and .Net 3.5 Framework

I came across this post in the SQL Server Slack chats and wanted to make sure that this information was shared so others don’t fall into this trap. Not having .NET 3.5 installed w/ SQL 2016 means no DB Mail and worse, no errors or alerts that it’s not working. I hope that MS will address this a bit more if they keep saying we don’t need .NET 3.5 for SQL 2016. In the meantime, if you want DB Mail, you’ll need the framework, too.

SQLSwimmer

There were so many cheers when Microsoft announced that the .Net 3.5 Framework was no longer a pre-requisite for installing SQL Server starting with SQL Server 2016. Folks were so excited, until they started testing certain pieces of the product, specifically Database Mail. That’s right, if you want to use Database Mail in SQL Server 2016 you have to install the .Net 3.5 Framework.

If you are installing SQL Server 2016 on Windows Server 2012 R2, this might be a problem for two reasons. First, the .Net 3.5 Framework is no longer included by default when spinning up a new install. Second, you will get no errors when testing Database Mail in SQL Server 2016. Mail will be queued, but will not be sent and there will be no errors in the Database Mail log, the SQL Agent log or even the SQL Server error log.

So if you’ve gone…

View original post 50 more words

Getting All Permissions for Server

We recently had to work out a way to fetch and display all effective permissions granted to our Production SQL Server. I had been required to get all permissions at the DB level before, but even then that tended to be more granular and only showed to what roles you belonged and what explicit permissions you’d been granted. We wanted to run something through a loop of all users and get that information into a report. It turns out this was easier than I’d expected, especially after my first attempts to dig through the system tables.

We used a script something like the following:

EXECUTE as login = ‘DomainName\Login’;
select ‘DomainName\Login’ AS Login, * from fn_my_permissions (NULL, ‘SERVER’);
REVERT;

With a little work this can be used within PowerShell or SQL to loop through all logins and capture the information in a table or report.

 

The function could also be used to loop through all databases by using the database and then substituting Database for Server above:

USE AdventureWorks2012;
EXECUTE as login = ‘DomainName\Login’;
SELECT ‘DomainName\Login’, * FROM fn_my_permissions (NULL, ‘DATABASE’);
REVERT;

For an interesting take on revamping sp_msforeachdb, check out this script from Edward Pollack (t). It might help you if you need to loop through all of your databases to run something like this for your auditors.

SQL Server Audits and Action_IDs

We were recently re-doing our SQL Server Audits and I was reminded again how painful setting the filters can be. MS expects an integer for “action_id”, but to actually use, them you need to know what those actually mean.

I came across this blog post by Chris Provolt listing the text versions of the action_id’s. That was helpful, especially the quick query to see what’s available:

Select DISTINCT action_id,name,class_desc,parent_class_desc from sys.dm_audit_actions

However, as you can tell by running this, the action_id’s returned are all text values. That doesn’t help when trying to set up your SQL Audits.

MSDN provides code for a function to translate the text audit_id into the expected numeric value.

CREATE FUNCTION dbo.Getint_action_id (@action_id VARCHAR(4))
returns INT
BEGIN
DECLARE @x INT;

SET @x = CONVERT(INT, CONVERT(VARBINARY(1), Upper(
Substring(@action_id, 1, 1))));

IF Len(@action_id) >= 2
SET @x = CONVERT(INT, CONVERT(VARBINARY(1), Upper(
Substring(@action_id, 2, 1))
))
* Power(
2, 8) + @x;
ELSE
SET @x = CONVERT(INT, CONVERT(VARBINARY(1), ‘ ‘)) * Power(2, 8) + @x;

IF Len(@action_id) >= 3
SET @x = CONVERT(INT, CONVERT(VARBINARY(1), Upper(
Substring(@action_id, 3, 1))
))
* Power(
2, 16) + @x;
ELSE
SET @x = CONVERT(INT, CONVERT(VARBINARY(1), ‘ ‘)) * Power(2, 16) + @x;

IF Len(@action_id) >= 4
SET @x = CONVERT(INT, CONVERT(VARBINARY(1), Upper(
Substring(@action_id, 4, 1))
))
* Power(
2, 24) + @x;
ELSE
SET @x = CONVERT(INT, CONVERT(VARBINARY(1), ‘ ‘)) * Power(2, 24) + @x;

RETURN @x;
END; 

Once you create that function, you can use it to generate a list of the Integer action_ids, the text action_ids, and the name/description of those action_ids. (You can also expand it out to see which actions are applicable to various objects at the server and database level if you so desire.)

SELECT DISTINCT dbo.Getint_action_id(action_id) Action_ID_Int,
action_id, NAME AS Action_Description
–,class_desc,parent_class_desc
FROM   sys.dm_audit_actions
ORDER  BY action_id

This will result in the following values (as of SQL 2012):

Action_ID_Int action_id NAME
1329873729 ACDO DATABASE_OBJECT_ACCESS_GROUP
542065473 ACO SCHEMA_OBJECT_ACCESS_GROUP
1329742913 ADBO BULK ADMIN
1346651201 ADDP DATABASE_ROLE_MEMBER_CHANGE_GROUP
1347634241 ADSP SERVER_ROLE_MEMBER_CHANGE_GROUP
538987585 AL ALTER
1313033281 ALCN ALTER CONNECTION
1397902401 ALRS ALTER RESOURCES
1397967937 ALSS ALTER SERVER STATE
1414745153 ALST ALTER SETTINGS
1381256257 ALTR ALTER TRACE
1280462913 APRL ADD MEMBER
538989377 AS ACCESS
1129534785 AUSC AUDIT SESSION CHANGED
1179866433 AUSF AUDIT SHUTDOWN ON FAILURE
1213486401 AUTH AUTHENTICATE
538984770 BA BACKUP
541868354 BAL BACKUP LOG
1111773762 BRDB BACKUP_RESTORE_GROUP
1179595331 C2OF TRACE AUDIT C2OFF
1313813059 C2ON TRACE AUDIT C2ON
1196180291 CCLG CHANGE LOGIN CREDENTIAL
1196182851 CMLG CREDENTIAL MAP TO LOGIN
1430343235 CNAU AUDIT_CHANGE_GROUP
538988355 CO CONNECT
538988611 CP CHECKPOINT
538989123 CR CREATE
538976324 D DENY
1179074884 DAGF FAILED_DATABASE_AUTHENTICATION_GROUP
1279738180 DAGL DATABASE_LOGOUT_GROUP
1397178692 DAGS SUCCESSFUL_DATABASE_AUTHENTICATION_GROUP
1178681924 DBAF DATABASE AUTHENTICATION FAILED
1396785732 DBAS DATABASE AUTHENTICATION SUCCEEDED
1128481348 DBCC DBCC
1195590212 DBCG DBCC_GROUP
541868612 DBL DATABASE LOGOUT
538987588 DL DELETE
1280462916 DPRL DROP MEMBER
538989124 DR DROP
541284164 DWC DENY WITH CASCADE
538990661 EX EXECUTE
538989638 FT FULLTEXT
541545542 FTG FULLTEXT_GROUP
538976327 G GRANT
1111773767 GRDB DATABASE_PERMISSION_CHANGE_GROUP
1329877575 GRDO DATABASE_OBJECT_PERMISSION_CHANGE_GROUP
542069319 GRO SCHEMA_OBJECT_PERMISSION_CHANGE_GROUP
1330860615 GRSO SERVER_OBJECT_PERMISSION_CHANGE_GROUP
1448301127 GRSV SERVER_PERMISSION_CHANGE_GROUP
541546311 GWG GRANT WITH GRANT
1346653513 IMDP DATABASE_PRINCIPAL_IMPERSONATION_GROUP
542133577 IMP IMPERSONATE
1347636553 IMSP SERVER_PRINCIPAL_IMPERSONATION_GROUP
538988105 IN INSERT
541214540 LGB BROKER LOGIN
1195525964 LGBG BROKER_LOGIN_GROUP
1094993740 LGDA DISABLE
1111770956 LGDB CHANGE DEFAULT DATABASE
1095059276 LGEA ENABLE
1279674188 LGFL FAILED_LOGIN_GROUP
1179207500 LGIF LOGIN FAILED
1397311308 LGIS LOGIN SUCCEEDED
1196181324 LGLG CHANGE DEFAULT LANGUAGE
541935436 LGM DATABASE MIRRORING LOGIN
1196246860 LGMG DATABASE_MIRRORING_LOGIN_GROUP
1296975692 LGNM NAME CHANGE
542066508 LGO LOGOUT
1146308428 LGSD SUCCESSFUL_LOGIN_GROUP
538988364 LO LOGOUT_GROUP
1111772749 MNDB DATABASE_CHANGE_GROUP
1329876557 MNDO DATABASE_OBJECT_CHANGE_GROUP
1346653773 MNDP DATABASE_PRINCIPAL_CHANGE_GROUP
542068301 MNO SCHEMA_OBJECT_CHANGE_GROUP
1330859597 MNSO SERVER_OBJECT_CHANGE_GROUP
1347636813 MNSP SERVER_PRINCIPAL_CHANGE_GROUP
1196182862 NMLG NO CREDENTIAL MAP TO LOGIN
538988623 OP OPEN
1111773263 OPDB DATABASE_OPERATION_GROUP
1448300623 OPSV SERVER_OPERATION_GROUP
1380013904 PWAR APPLICATION_ROLE_CHANGE_PASSWORD_GROUP
541284176 PWC CHANGE PASSWORD
1195595600 PWCG LOGIN_CHANGE_PASSWORD_GROUP
1396922192 PWCS CHANGE OWN PASSWORD
1480939344 PWEX PASSWORD EXPIRATION
1129142096 PWMC MUST CHANGE PASSWORD
1280333648 PWPL PASSWORD POLICY
542267216 PWR RESET PASSWORD
1397905232 PWRS RESET OWN PASSWORD
542463824 PWU UNLOCK ACCOUNT
538976338 R REVOKE
538985298 RC RECEIVE
538986066 RF REFERENCES
538989394 RS RESTORE
541284178 RWC REVOKE WITH CASCADE
541546322 RWG REVOKE WITH GRANT
538987603 SL SELECT
538988115 SN SEND
1313624147 SPLN SHOW PLAN
1448301651 STSV SERVER_STATE_CHANGE_GROUP
1313953107 SUQN SUBSCRIBE QUERY NOTIFICATION
1313035859 SVCN SERVER CONTINUE
1146115667 SVPD SERVER PAUSED
1146312275 SVSD SERVER SHUTDOWN
1381193299 SVSR SERVER STARTED
1095975252 TASA TRACE AUDIT START
1347633492 TASP TRACE AUDIT STOP
538988372 TO TAKE OWNERSHIP
1111773012 TODB DATABASE_OWNERSHIP_CHANGE_GROUP
1329876820 TODO DATABASE_OBJECT_OWNERSHIP_CHANGE_GROUP
542068564 TOO SCHEMA_OBJECT_OWNERSHIP_CHANGE_GROUP
1330859860 TOSO SERVER_OBJECT_OWNERSHIP_CHANGE_GROUP
1195594324 TRCG TRACE_CHANGE_GROUP
542069332 TRO TRANSFER
1346847573 UCGP USER_CHANGE_PASSWORD_GROUP
1195459669 UDAG USER_DEFINED_AUDIT_GROUP
1430340693 UDAU USER DEFINED AUDIT
538988629 UP UPDATE
1178686293 USAF CHANGE USERS LOGIN AUTO
1196184405 USLG CHANGE USERS LOGIN
1129599829 USTC COPY PASSWORD
1414743126 VDST VIEW DATABASE STATE
1414746966 VSST VIEW SERVER STATE
1413699414 VWCT VIEW CHANGETRACKING
538984792 XA EXTERNAL ACCESS ASSEMBLY
538989912 XU UNSAFE ASSEMBLY

Database Lifecycle Management – Dacpacs

I came across an excellent article recently @ Simple-Talk discussing Dacpacs, working with them, structures, etc. in quite a bit of detail for a short article. This post discuses benefits, drawbacks, and potential areas of concern for working with dacpacs for your database management.

https://www.simple-talk.com/sql/database-delivery/microsoft-and-database-lifecycle-management-%28dlm%29-the-dacpac/

The PowerShell scripts are well worth checking out and there are several interesting calls for SQLPackage along the way. I appreciated his findings with some of the issues – not always seeing changes that should be detected. I suspect that there’s a setting not configured properly there, but couldn’t say for sure. I know that I haven’t hit any issues when using SQLPackage and dacpacs for normal DB development of the sort discovered at the end of the article.

The conclusion is spot on, whatever solution you use – check your scripts, test, and verify before just putting the solution into production.

Pre-deploy SQL Scripts before checking for changes

I just came across a post from Ed Elliot (b | t ) about running Scripts before the SQL Package process runs. Sort of a “Pre” Pre-deploy script.  The concept is simple enough, but it’s something people have wanted for a while.

Here’s the main article.

The general idea is that you add a SQL Script to your project with:
Build Action = “None”
Copy To Directory = “Copy Always” (or “Copy if Newer”)

When you set up your publish actions, look for this script first, run it in PowerShell, SQLCMD, or whatever other option you might have, then run your SQLPackage commands as normal. It will run through whatever scripts you want to run first, then compare against the modified database to bring it up to date.

The idea is pretty clever, though I haven’t come across a lot of need for it when trying to deploy DB changes. I’m not quite comfortable with a script running to change the state of the database to something unknown prior to running the script to bring it in line with version control. I suppose it has some use-cases, though, as I’ve come across this request several times on Stack Overflow.