Redgate Data Catalog DB Documentation

• 
  Many thanks for this, Dave. I'll pass this onto the Product Team for Data Catalog, who I'm sure would be really interested in seeing them.

  Roseanna April 01, 2020 09:22

  - 0 +
• 
  Thanks Roseanna.  One possibility is that I put the scripts in my own GitHub repo and let the team pull from it.

  Dave60103 April 06, 2020 11:33

  - 0 +
• 
  @Dave60103 great to hear from you again so soon! Yes, we would definitely be interested in the scripts and we have repo (though we should really do more to promote it with our collaborators in due course).
  
  I'm dropping an email over to you now so @toby.smyth  Tech Lead for Data Catalog can guide you to our repo and kick off some collaboration. Look forward to seeing how this progresses! A

  Andy_Richardson April 07, 2020 16:33

  - 0 +
• 
  https://github.com/AspieArchitect/DataCatalogDoc
  
  I've burned a lot of time using Powershell to amend the output of SQLDoc as a post processing step.  To quote Edison, I have not failed, I have found 10,000 ways that do not work.  An article I hoped to complete months ago is in tatters.
  
  The challenge I have is that I like to use different ways of drawing attention to database documentation content so that it is easy to read and useful to the reader.  The reader is afterall the most important person.
  

  • Bulleted lists
  • Numbered lists
  • Hyperlinks to external documents i.e. design documents on Atlassian Confluence, articles on SQLServerCentral or Simple-talk etc.  The more connected the content the more useful it is to the reader.

  The MS_DESCRIPTION property is displayed in SQLDoc, Data Catalog and SQLPrompt as tooltips.
  
  Any attempt to use HTML to support this is rebuffed as it converted to html encoded text.  I did experiment with using Powershell to modify links stored as Markdown format into html format.
  Similarly markdown bulleted lists converted into html.
  
  If I use markdown bulleted lists for in column descriptions then they show in html and tooltips (albeit with * for bullet) but break markdown output in markdown tables.
  
  If I use markdown hyperlink format I can reformat the link so it renders in html output but the tooltips in SQLPrompt become cluttered with markdown.  Their clarity is lost.
  
  I almost had a solution but not one that was robust enough for my liking.  Even using the Atlassian products directly (JIRA, Confluence, BitBucket) you have to be vigilant for things that break text formatting or macros.  As MS_DESCRIPTION is effectively free text it is too easy to break the content.
  
  I feel that the facility to support lists and hyperlinks in database documentation would be useful for the reasons stated.
  In order for it to work the rendering of tooltips in SQLPrompt would have to discard any presentation mark up (HTML or MD) or render it somehow.
  
  Rendering acceptable content in HTML is simplest.
  
  I haven't tried the other supported output formats of SQLDoc.
  
  When using markdown format I switched all instances of index.md and references using index.md to readme.md.  By default git based repos automatically render readme.md. They don't render index.md.
  
  I also found that I had to copy some readme files so that access to any folder in a git based repo would have an appropriate readme.
  
  I'm frustratingly close to getting all this working, but I've been frustratingly close for ages.  Some of it is due to lockdown burnout.

  Dave60103 June 28, 2020 17:48

  - 0 +
• 
  @Dave60103 - In your specific case, who are the readers? ie, what problem the reader expect to solve that is aided by reading the documentation?

  David Atkinson June 29, 2020 08:27

  - 0 +
• 
  @David Atkinson sorry for the delay.
  In our case it is primarily the data scientists and the data analysts who need to know what the data structures are supposed to hold.
  
  As data governance becomes ever stricter the data compliance folk are taking an increasing interest as well.  Every year we get an external data audit and the questions we have to answer get more detailed as time goes on.  What data do we have, how can you prove data starting at point A gets to point B and beyond etc.
  
  Another problem we are trying to solve is that a development team are not sure what a database column is for and rather than try and find out they simply create another column.  We had a case where a web app was failing intermittently and what we found was the practise of bunging another column on the end of a table meant that in some cases the app was attempting to write a record greater the 8060 bytes.

  Dave60103 July 21, 2020 16:41

  - 0 +
• 
  @Dave60103
  Thanks. So in summary the use cass in order of importance right now are:
  1) as a data dictionary reference for data scientists, data analysts
  2) to support a data audit (can you elaborate on what you mean by proving that starting point A gets to B?)
  3) as a data dictionary reference for developers

  David Atkinson July 22, 2020 09:03

  - 0 +
• 
  Yes, that's it.  I would be interested in other people's perspectives on this. 
  My experience has been that as the data estate grows you can no-longer rely on the hive mind of the organisation.  COVID-19 has really disrupted the hive as well.

  Dave60103 July 23, 2020 07:38

  - 0 +
• 
  @David Atkinson I have finally got the code and article together to describe how I extend Redgate SQLDoc to support
  

  • Hyperlinks in HTML
  • Bulleted lists in Git markdown table cells
  • README.md as the default document rather than index.md
  • README.md in each object folder (Tables.md, Views.md etc get copied to README.md in that folder)

  Normally I would submit the article through Tony Davis.  It has been a long time since I submitted anything so I am not sure if this has changed?
  
  

  Dave60103 December 18, 2020 15:40

  - 0 +
• 
  @Dave60103 - Sounds great. Tony is still a good point of contact for this sort of article! 
  
  

  David Atkinson December 18, 2020 18:00

  - 0 +

Add comment

Please sign in to leave a comment.