SSIS Catalog (SSISDB) Cleanup

I really appreciate what MS has done w/ the SSIS Catalog. We have built-in logging at a level that wasn’t possible in prior releases, but that comes at a cost. The default retention is pretty high (365 days) and MS chose to handle cleanup using cascading deletes. This sort of coding makes life easier for the developers, but almost always performs poorly. That’s especially the case when you have 1 parent row with 100’s of thousands or child rows related to it.  The problem is compounded because the first time people realize they need to lower the retention limit is about the same time that the database is starting to fill up or has filled up. At that point, it’s too late to lower the retention by a large number because the corresponding delete will still cause issues.

I knew someone had written some code for this and a quick search on StackOverflow found this question and the answer from Bill Fellows ( b | t ) :
http://stackoverflow.com/questions/21781351/how-can-i-clean-up-the-ssisdb

I’m including the code here for a quick reference, though I recommend reading his answer, up-voting it, and checking out his other resources. The only change I’d consider making is perhaps clearing out the child tables in batches of 1000-5000 rows at a time instead of all rows. I found that I had to work in increments of no more than 10 days at a time to delete history. Any more than that and the time to execute increased quite a bit. In this case, look for your max days currently retained, and drop the @DaysRetention variable to that less a couple of days to see what is removed. Keep dropping that until you reach the point at which you want to stop, set your SSIS Catalog retention days to that value, and you should be good from that point on.

It’s probably worth noting that in SQL Server 2012, MS didn’t include proper indexing by default. You may need to add some indexes in order to have the delete perform in a timely manner and avoid excessive table scans. The index issue was addressed in SQL Server 2014 and higher (and I think in SQL Server 2012 SP2 as well).

Advertisements

4 thoughts on “SSIS Catalog (SSISDB) Cleanup

  1. Pingback: Cleaning Up SSISDB – Curated SQL

  2. Have you considered, or seen an example of, developing a secondary cleanup procedure that thins out the logged data after an intermediate period? I’d like to keep 370 days of retention, as there are some packages that only run annually, but after a short time, 1 or 2 weeks, I’m really only interested in post execute events, errors and the overall run time. A second procedure that thinned out the data after the 1 or 2 week period could hopefully reduce the size of this database drastically.

    Like

    • I haven’t looked into it myself, but I imagine you could tweak the above script to handle removing the data you want to remove. You could filter out for the packages you want, set the retention to something crazy high to keep everything you need, but still keep what you want. Just set this up as its own scheduled job.

      Like

  3. Thank you for the reply. I agree that it doesn’t appear to be technically difficult, and with the way that the built in cleanup appears to be built on cascading deletes I believe that thinning the related data should actually speed that part up. I do wonder what the impact will be on the built in reports, but I’ll give it a shot in a test environment to see how it looks.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s