PowerShell and SubVersion with XML

Recently I was asked if it was possible to automate part of our release process whereby our release management department needed to find what SQL scripts in a particular database directory had changed between specific repo revisions on a specific path. It is possible to automate the above and I have chosen PowerShell using the SVN client binaries (which output XML that can be easily processed) to achieve this. I have listed an example of the XML skeleton returned by SVN and the majority of the PowerShell code which will collect the SQL file URLs - once you have the URLs, they can be exported.

So, the SVN client binaries can output, in XML format, which files have been modified on a certain path between certain revisions and PowerShell can process XML using the Select-Xml cmdlet with some XPath. To generate an XML document which can be piped to the PowerShell Select-Xml cmdlet, execute a command similar to the following:

svn.exe log http://path/to/do/an/svn/log/on --verbose --xml --revision 4453:4478

The above will return an XML document ready for piping - the following is an example of the skeleton. It contains two SVN logentries (one logentry = one commit).

<?xml version="1.0"?>
<log>
    <logentry revision="4467">
        <author>jdolan</author>
        <date>2012-02-24T17:06:46.686349Z</date>
        <paths>
            <path
               kind="file"
               action="M">[/project_name/etc]</path>
            <path
               kind="file"
               action="M">[/project_name/etc]</path>
        </paths>
        <msg>[the_commit_message]</msg>
    </logentry>
    <logentry revision="4488">
        <author>ccallaghan</author>
        <date>2012-02-29T12:36:10.386087Z</date>
        <paths>
            <path
               kind="dir"
               copyfrom-path="[/project_name/etc]"
               copyfrom-rev="4566"
               action="A">[/project_name/etc]</path>
        </paths>
        <msg>[the_commit_message]</msg>
    </logentry>
</log>

The pseudo-code for the PowerShell script below is:

• Get an XML document representing all SVN logentries on a particular path between particular revisions and pipe it to...
• The Select-Xml cmdlet which uses XPath to get a per logentry and pipe it to...
• The ForEach-Object cmdlet which iterates over each logentry and allows for the processing to take place. Here we can obtain the revision, if the file has been modified, added or deleted, the commit date, the path to each file and even the log message. After some processing, we can add to a list the fully qualified paths to the SQL files we are interested in.

You will need to fill in the appropriate blanks, such as the root and log URLs, to revision, from revision, the file extension to match, etc - for convenience, I have declared these for you at the top of the script below. As the script is well commented, I'll let it speak for itself rather than dissecting it too much

$fromRev = 4453
$toRev = 4478
$svnUrlRoot = "http://root/path/to/repo"
$svnUrlLog = "$svnUrlRoot/path/to/do/an/svn/log/on"
$fileExt = ".sql"

# Using the svn log command, get an XML document representing the preset criteria 
# and cast it to an XML object, pipe this to the Select-Xml cmdlet to get the 
# logentries and pipe these to the ForEach-Object cmdlet to iterate over them.

([xml] (svn.exe log $svnUrlLog --verbose --xml --revision ""$fromRev"":""$toRev"")) |
        Select-Xml -XPath /log/logentry |
        ForEach-Object {
            $logEntry = $_
            $logEntryPaths = $logEntry.Node.paths.path
           
            # Iterate through each path in the logEntryPaths XmlElement.
            foreach ($logEntryPath in $logEntryPaths)
            {                   
                # Get the contents (relative HTTP path) of the logEntryPath and append it to 
                # the SVN root URL. #text must be in double quotes to ensure the hash / pound symbol is 
                # correctly dealt with by PowerShell.
                $sqlFileUrl = ($svnUrlRoot, $logEntryPath."#text") -join ""
               
                # Ensure that URL's matching the component database 
                # pattern and ending with .SQL are examined.
                if ($sqlFileUrl -imatch $sqlFileUrl.EndsWith($fileExt))
                {
                    $fileAction = $logEntryPath.action
                   
                    # If the SQL file has been deleted, it should be ignored 
                    # as it no longer exists.
                    if (! ($fileAction -ieq "D"))
                    {
                        # Determine if the SQL file already exists
                        # in the array from a previous relevant rev,
                        # if not, add the complete URL to the array
                        # for export later. This prevents
                        # duplicates as a developer may have made
                        # many commits to the same SQL file. 
                       if ($sqlFileList -inotcontains $sqlFileUrl)
                        {
                            $sqlFileList += $sqlFileTagUrl
                        }
                    }
                }
            }
        }

Now that we have a list called $sqlFileList containing the paths to all the fully qualified SQL files, we can put it into a foreach loop and using the svn export command to export them to our desired location.

If using the standard trunk, tag, and branches repo layout, remember that if executing this script on tag, the paths which the svn log will return, will probably be from trunk so if you wish to export the SQL files from the tag, you will have to replace part of the URL from trunk to tags, before adding the URL to the export list.
       
We also had a requirement whereby we would like to ignore any SQL files which were forward merged. This can be achieved in a variety of ways but we have found the easiest to be the following: ensure there is a standard commit message for forward merging and in the above powershell script, extract the log message per logentry and compare it to the pre-defined message and if it matches, ignore the contents of that revision.

As an aside, much of the above code can be reused to read and process any XML document in PowerShell. Consider replacing the svn log command line above with the Get-Content cmdlet which allows you to read the document from a file instead of the output returned from a command. The XPath and SVN specific logic can also be replaced. Another useful test before processing XML is ensuring the XML element has child nodes. To do this in the above example with logentry, add an if statement that evaluates $logEntry.Node.HasChildNodes which was omitted as the svn log XML will always have child nodes.

SubVersion BAT / batch pre-commit hook to exclude specific directories

[2012-04-27] Updated based on user feedback: thanks to Alex & Dan.

Recently, I have encountered an issue where I found that Microsoft .NET bin and obj directories were being committed to our codebase. These directories are compilation output directories used by MSBuild / Visual Studio and thus should not be added to the SubVersion codebase. In an effort to stop this, writing a pre-commit hook was the obvious answer. However, I have chosen to write it in batch / BAT.

Yes, BAT files, I know what you are going to say... Why not Perl, Python or even some BAT file that calls out to Powershell (which is my favourite scripting language)? Well, the simple answer is speed. With time constraints in place, getting Perl and Python interpreters set up on the SubVersion server with a language learning curve for our team may not be justifiable for a single pre-commit hook where a quickly written BAT file would suffice. Initial investigation suggests that getting Powershell running with a pre-commit hook BAT wrapper may prove too time costly also, so batch was chosen. This entry assumes a basic familiarity with BAT files.

OK, some interesting points of note about SubVersion pre-commit hooks. 

• Because this is a pre-commit hook, it does not deal with revisions, it deals with transactions, which, may or may not be persisted becoming revisions, depending on the outcome of the pre-commit hook. The format of transaction IDs differs compared to revisions IDs.
• If the pre-commit hook exits with a return code of 0, then the transaction is persisted and becomes a revision. If the pre-commit exits with a non-zero return code, the transaction will fail to be committed. There is scope here to provide a specific message which is sent back to the user, detailing why the hook failed which can be seen with each of the four echo commands below and how that command's standard output is redirected to the standard error via the >&2 code.
• When the BAT pre-commit hook below is executed, there is no environment set - such as the system path, so when using the svnlook executable, the absolute path must be provided. Similar for other commands. Don't forget to wrap paths in double quotes if they contain spaces!

The example below prevents commits which include empty bin and obj directories and bin and obj directories which contain one or more files / folders. The core of the example is using svnlook to examine what paths have changed on the current transaction and piping these to the Windows findstr command which uses a regular expression and checks to see if bin and obj directories exist on those paths.

Finally, to add this to SubVersion for a specific repo, navitate to the repo on the file system, and under the hooks directory, create a new file called pre-commit.bat and add the script below to it.The only update you'll need to make to this script to get it working is to set the correct path to the svnlook executable in each of four locations below. Save the file. The moment this file exists and the contents are valid, the pre-commit hook is live on the repo.

:: Set the first incoming parameter to the absolute path to the repository and the second incoming parameter to the transaction ID. This is standard in all SubVerison pre-commit hooks.
SET REPOS=%1
SET TXN=%2 

:: Check if there is an attempt made to commit changes to a bin directory. This also includes an attempt to commit an empty bin directory.
"E:\csvn\bin\svnlook.exe" changed -t %TXN% %REPOS% | FINDSTR /R /I "./bin/.*"
IF %ERRORLEVEL% EQU 1 GOTO CHECK_OBJ_DIR
ECHO "It is forbidden to commit empty bin directories or to commit to existing bin directories to SubVersion as these are generated .NET output and not part of the SubVersion version controlled codebase" >&2
EXIT 1
:: Check if there is an attempt made to commit changes to an obj directory. This also includes an attempt to commit an empty obj directory.
:CHECK_OBJ_DIR
"E:\csvn\bin\svnlook.exe" changed -t %TXN% %REPOS% | FINDSTR /R /I "./obj/.*"
IF %ERRORLEVEL% EQU 1 GOTO OK
ECHO "It is forbidden to commit empty obj directories or to commit to existing obj directories to SubVersion as these are generated .NET output and not part of the SubVersion version controlled codebase" >&2
EXIT 1

:: All checks passed, so allow the commit.
:OK
EXIT 0

Considerations when Changing Jobs

[This post has been updated as of 2013-03-26]

While this is a little off topic, I think this could be useful for people considering changing job. It started as a list of considerations which I casually discussed with a friend but eventually grew to what is detailed below. I have also updated it with experiences discussed with other people I know. It assumes that you are happy enough in your current sector. First, a bit about me to give my musings context. I'm in my late twenties, and was lucky enough to work for a good independent software vendor straight after college, spending six years there before changing jobs in August 2011 to another company. The items are not in any particular order as the priority will differ depending on one's lifestyle, family, personality, circumstances, etc.

Salary. Generally, if a move is not based on a lifestyle choice and the responsibilities are similar or greater with the new role, it stands to reason that the new employer will pay more. A great place to start estimating salary is on a a jobs / recruitment site.

Experience. This can be more important than salary when looking for a challenge, starting out a career or looking to move sideways in a career. Sometimes, it may become damaging to a career to continue to stay in a company with few challenges and / or where technology is somewhat stagnant or completely bespoken to said company. A new experience also helps to fight the inevitable institutionalism with which can occur after many years. Another big part of this is if the work in the new company appeals more so than the old.

Corporate culture. Which is always hard to judge because at the interview, a glimpse is all that is really gotten of how things work. Knowing somebody already in the company or having a friend or a friend of a friend who works there can help. Failing that there's plenty of information online about the larger companies and how they function. If possible, ask to be shown around the office to get a better feel of how things hang together.

Work ethic. Tied to corporate culture and differs from person to person and place to place. Also, just because the offices are based in one region, doesn't mean it will inherit that localities work ethic. 

Career progress. Possibly part of the reason for wanting to leave a current employer. Sometimes a move upwards in salary is a move upwards in responsibility, sometimes it's just a move sideways. Career progress in a company is something which should be seriously considered before the interview.

Job security. An increasingly important point. Check if the company is public or private and if it Irish owned or is it a subsidiary or similar of some conglomerate. There are pros and cons to each of the above which are beyond the scope of this post.

Socialising. Not to be underestimated. People make a company and great people make companies great. A company which has an active sports and social committee is generally a good sign. Check the average age group of the company compared to the current company - it can be a somewhat daunting starting out in a company where most are ten years or more senior, many have families and are settled.

Number of working hours. This is a niggley one and usually falls somewhere between 37.5 and 40 hours for much of the IT sector. Also note that it's important to find out if it will be expected to work overtime and if so, what the compensations is - money, days in lieu or nothing at all. I've found that the corporate culture and work ethic have quite a role to play in this too. It was only after an offer was posed to me but before I accepted that I realised that I would be working an addition 2.5 hours per week. This must be factored into a pay rise - ie: if the hours are greater, then subtract from the rise and vice versa. Again, for those with a family or thinking of starting one, the extra time could make an impact either getting up earlier or working later.

Holiday allowance. The number of days leave per annum, and number of days per years of service. While losing holidays when changing jobs does not equate to a lot of money financially, still equates to time off. Also, it may be possible to buy days or take unpaid leave or that the company give personal days or that days can be worked up in over-time.

Location. Check in detail where the company is based and if it is convenient to get to and socialise from. Also, check for places of interest around the company such as places to eat or gyms etc. if necessary. While many of these are relatively trivial issues, when they start to negatively come together, it can make for some serious consideration.

Healthcare. This is becoming important given the increasing cost of it this year (2012), especially for families and a subsidised or complete company healthcare scheme is certainly valuable.

Pension. Most of us will get old and probably live longer than the current life expectancy - pensions are important! Remember that if a company does not have a pension fund and do not contribute towards private funds, you must factor this into a pay rise. Also, in a company like this, there may be more cost associated with a contribution as it may come out of your net pay rather than gross and so be worth much less.

Family benefits. Family oriented benefits past the legal minimum such as additional days off for dads, an on-site crèche, inclusive healthcare, opportunity to negotiate taking a day off a week for several months while rearing children - these are all very convenient.

Flexi-time. Useful to have, especially with regard to a family oriented lifestyle - picking kids up from school, for example. 

Working from home. This is another benefit that can be very useful to have, especially when there may be a long commute involved, distracting working conditions and other more family oriented reasons.

Hardware. For a person working in the software industry, a fast computer and dual screens are invaluable and should be the de facto. Another point to note is that the computer should be clean - as in re-ghosted or reinstalled from scratch - not somebody's hand-me-down with all sorts of stuff on it. Even a proper ergonomic chair, keyboard and mouse are great, considering the sheer amount of time spent sitting down, typing and clicking

Perks. Which can include Christmas bonus, company performance bonus, personal performance bonus, quota bonus, company car and many others.

Probation. While this might not be normal for all, it certainly applies in the software sector in Ireland and when on probation, the employee won't have their healthcare paid for or benefit from a company pension contribution which should be taken into consideration. This means that for many months, a little pay cut will have to be taken to pay for healthcare and consideration given as to what to do with an existing pension before moving it to the company's pot.

Title. Preservation of existing title may be important to you. You may be a Senior Build and Release Engineer and a company you wish to move to may be just looking for a Build Engineer and while the role may be similar or have greater responsibility and remuneration, the title may not reflect this.

Freedom of expression. having the freedom to express one’s self and opinions is very important, especially given all the mediums which make information so darn accessible these days. Writing code outside of work hours (perhaps contributing to some open source project, designing websites, etc?) and blogging is great fun and very rewarding but what exactly does the contract say about this? What is the company's position on this? Perhaps nothing at all or perhaps it is quite restrictive, making it more difficult to be yourself.

Notice. In IT, giving the company notice of resignation is usually one month in advance of physically leaving. However, depending on the company and the amount of responsibility the role contains, this may be extended. Do not assume that it is one month! Ensure you know what what your notice period is and that it suits you. It may be easier to move to a new company when the notice period of your current company is one month but if it was two months, it might be too big a gap for the prospective employer to wait, especially if you assumed it was one month and only found out it was two upon handing in your notice!

Requirements / comportment regarding Twitter, Facebook, MySpace and the plethora of other social networking sites. Does the contract mention these sites, one's conduct, a requirement for an official work account, etc? Company's are becoming ever more vigilant about how they are perceived, especially online and may have rules about what sort of comments their employees can make about them via the above media. They may also encourage / require you to open / maintain an work account with one or more such sites - I'm unsure of the legalese here.

Non-Compete / Anti-Compete / Covenant-Not-To-Complete Clause. This is where an employee agrees not to work in a new but similar role which competes (at least in the employers view) against the their past employer for a specified duration such as 6 months. This is one to look out for, especially when it comes to the ICT sector. Be wary of clauses such as this innocuously appearing in the contract without the prospective employer or recruiter drawing attention to it. 

Pre-booked holidays. Don't forget to mention any holidays which you may have already pre-booked as it would be a nasty surprise to change job, only to find out that that summer holiday which you were looking forward to may no longer be possible. 

The contract. Read it carefully and compare it with your current role and what has been promised in the new role, especially with regard to working hours, notice, social networking comportment, non-compete/anti-compete/covenant-not-to-compete clauses. Ensure a printed and signed copy of the contract is mailed out, ideally before starting in the new company. A physical contract can be very important later on, as for example, if the role is being made redundant and there is a redundancy entitlement - if no signed physical or even virtual contract exists, then it becomes hard to prove one's entitlements. 

Finally...  Just some general advice - be mindful (love that word :) of what items above are discussed during the interview and what are discussed after an offer is made but before the offer is accepted. The former is a position with much less bargaining power than the latter - know what to ask in the interview and what to negotiate after the offer but before accepting. I will add to this post when I think of / discover any additional, useful information and perhaps tighten it up a bit over time.  A wise friend once said that a change is as good as a break - he was right.