Corporate Powershell Module Repository – Part 1 – Design and Infrastructure

Modules – Modules – Modules

If you’ve ever had a Powershell conversation with me the chances are high that you’ve heard my agenda around modules.  I think that the 2.0 module cmdlets were exactly what Powershell needed.  I come from the land of Perl where modules are an integral part of the community.  Well-formed reusable code is something Perl developers take a lot of pride in.  Unfortunately, the Powershell community has been very delinquent in picking up the Microsoft implementation of modules.  This is mainly due to the lack of a well-designed Internet repository that users can easily install and import from.  While this is a difficult task that we have yet to take on in the Powershell community, it is a much easier task to take on in a corporate environment.

Unfortunately, I can’t disclose the name of the company I work for, but I can tell you it is very large with about 100,000 users and plenty of IT folks to support it.  When I started there three months ago I was extremely surprised at how amazing their Perl repository and documentation was complete with wiki sites, in-house modules, and distribution of those modules through a globally distributed file system.  This was a fountain of information to drink from.  Then I went to the Powershell site: ugh – a well of cyanide!  The home page still referred to it as Monad!  The snapins that were installed to the distributed file system were versions behind, and the documentation really needed an overhaul that reflected the usage of both Powershell 1.0 and 2.0 together.

It was a daunting task, but I quickly volunteered to take it on.  Sweating nights and weekends I not only redesigned the documentation, but I put together a module repository that I am extremely proud of.  Here’s what I did:

Gathered Requirements

I first spoke with users of Powershell in the company to understand how it’s used, and to understand what ways a module repository could be helpful.  I came up with the following requirements:

1. The repository had to support multiple versions of the same module
2. Users who were using Powershell as a shell expected to have the most recent version of a module loaded when using Import-Module
3. Users who were writing scripts needed the ability to specify a version number of the module to use in order to ensure that a new version of a module does not break their script.
4. The repository needed to be available globally by all workstations and servers
5. The repository required robust documentation for each module.

I had one additional requirement that came from me:  Use the built-in cmdlets for modules i.e. Import-Module and Get-Module

Leveraged Existing Infrastructure

The distributed file system they use is extremely custom and is AFS based.  While my design could use any CIFS share it was nice to have something with such strict control.  I can publish to a \dev workspace for testing.  Users only have access to DEV if they have their machines configured to use it via the registry.  After testing and development is complete I can lock the DEV instance of a module and publish the files to a read-only file system accessible by all users and servers in the company around the world.  The read-only piece is key.  If you wind up configuring your in-house module repository using my method I highly recommend making the repository read-only with only a few key accounts having access to publish.

For the remainder of this article we will call the published read-only share: \\powertoe\modulerepo\

Configure the Code Access Security Policy

This is the only piece that requires administrator privileges.  Because the distributed file system we use is already leveraged heavily by other .NET groups within the company this part was already in place in our desktop and server builds.  In order to allow .NET to trust a share so it can load .DLLs from it you need to modify the policy with caspol.exe.  Caspol.exe can be found in your .NET directory, and documentation about it’s usage can be viewed here.

Here’s a quick batch script that will determine the proper directory to run caspol.exe from, and then call caspol.exe with the proper parameters to configure our system to trust the \\powertoe\modulerepo share:

@echo off
setlocal
for /f %%a in ('dir %windir%\Microsoft.Net\Framework\v* /b') do @call :caspol %windir%\Microsoft.Net\Framework %%a
for /f %%a in ('dir %windir%\Microsoft.NET\Framework64\v* /b') do @call :caspol %windir%\Microsoft.Net\Framework64 %%a
endlocal
goto :EOF
:caspol
set sc_dotnetpath=%1
set sc_dotnetversion=%2
%sc_dotnetpath%\%sc_dotnetversion%\caspol -polchgprompt off
%sc_dotnetpath%\%sc_dotnetversion%\caspol -m -ag 1 -URL "file://///powertoe/modulerepo/*" FullTrust
%sc_dotnetpath%\%sc_dotnetversion%\caspol -polchgprompt on
goto :EOF

This step is only necessary if you are either using snapins that you will convert to modules or if your modules load dlls of any kind.

Design the Folder Structure

This was a big hurdle to overcome.  Because I needed to nail the versioning of modules for many different purposes I spent a lot of time toying with using the -version parameter of Import-Module.  What a dead end that was!  The -version parameter seems to have been an afterthought to versions: Documentation that doesn’t work and a backwards method of selecting the highest version of a module frustrated me to no end.  You can read a little more of this digression here.

After throwing away “Import-Module -version” I came up with an elegant solution to the problem.  I decided I could script the user’s $env:PSModulePath.  If you are not aware of how modules become available to the built-in cmdlets Import-Module and Get-Module you should read:

Get-Help about_modules

To summarize the relevant parts: the environment variable PSModulePath holds the file paths that the built-in cmdlets look in to find modules that are available.  For example if the PSModulepath is set to c:\modules and you have a directory in that path called ToeISE with a ToeISE.psm1 or ToeISE.psd1 file it will be an available module when you run:

Get-Module -ListAvailable

If it is in that list you can then load the module with the following:

Import-Module modulename

As I mentioned I did struggle with trying to use a folder structure like \\powertoe\modulerepo\modulename\version or even \\powertoe\modulerepo\modulename\version\modulename with the $env:PSModulePath pointing to \\powertoe\modulerepo, but nothing worked quite the way I needed it to.  In the end I used the following configuration:

\\powertoe\modulerepo\modulename\version\modulename

Design the Profile

To fix the PSModulePath to point to the latest version of every module in the repository I added the path of the latest module to $env:PSModulePath through a quick one-line script:

Get-ChildItem \\powertoe\modulerepo\|where{$_.psiscontainer}|foreach {Get-ChildItem $_.fullname|Where{$_.psiscontainer}|sort -Descending|select -First 1}|foreach {$env:PSModulePath += ";" + $_.fullname}

or for better readability:

Get-ChildItem \\powertoe\modulerepo\|where{$_.psiscontainer}|foreach {
    Get-ChildItem $_.fullname|Where{$_.psiscontainer}|sort -Descending|select -First 1
}|foreach {
    $env:PSModulePath += ";" + $_.fullname
}

In the end, however, I wrapped this one line in a script and published it to \\powertoe\modulerepo\modulerepo.ps1.  I documented that users should dot (.) source this script in their profile.  It’s a similar copy/paste operation, but it allows me to control the profile of our users if anything in the module repository ever needed to change.

. \\powertoe\modulerepo\modulerepo.ps1

There is one small problem with my method:  It may not scale very well.  As the module repository grows it is possible that the script that loads PSModulePath will be too cumbersome.  If that ever happens I can modify modulerepo.ps1 to include the full path for each module.  It would then be a part of the controlled distribution process of any module to update this script with the new version of the module uploaded.  Rather than introducing a manual point of failure I decided to tackle this problem if the module repository ever grew large enough where it would impact performance.  There is also the question of whether performance will ever be hindered by a PSModulePath that becomes too large.  Again, a problem I am resolved to tackling down the road if it ever manifests.  In the meantime, the above solution gives our users a nice way to use get-module -listavailable and import-module modulename to use the central repository.  This works for the latest version of any module, but it doesn’t handle how to specify a version to use.

Controlling Versions in Scripts

The solution is really a no-brainer.  Import-module supports the usage of paths with the -name parameter.  So simply users are instructed to use the full path to the .psd1 in their scripts:

Import-Module -name \\powertoe\modulerepo\PowerCLI\4.1.0\PowerCLI.psd1

Because we control what gets distributed to the module repository we can enforce standards e.g. Must have a psd1 file, must have inline help with specific fields filled out, must use proper namespaces, etc.  Perhaps I will share that bit of information at a later date with you, but I know that there’s something that will interest you much more than an article about best practices.  If you read the above example for Import-Module you’ll see that I’m using it on a .psd1 file for PowerCLI – Wait a minute – Does that exist?  Does that work?  Can anyone guess what I will writing about in a future article?

Recap

• Create share and publish modules to \\server\share\modulename\version\modulename
• run caspol to trust \\server\share
• Use the one-liner in your $profile:

Get-ChildItem \\server\share\|where{$_.psiscontainer}|foreach {Get-ChildItem $_.fullname|Where{$_.psiscontainer}|sort -Descending|select -First 1}|foreach {$env:PSModulePath += ";" + $_.fullname}

• Use the following to view available modules:

Get-Module -listavailable

• Use the following to import modules from the repository:

Import-Module modulename
or
Import-Module -name \\server\share\modulename\version\modulename\modulename.psd1

Modules – Modules – Modules

They are a cool powerful way of sharing your well-formed code.  It is a great way to promote best-practices internally and enforce your policy on reusable code.  They are also an amazing utility that lets you quickly browse and install other people’s code.  I strongly suggest you start using them.  Next stop, let’s do this on the Internet people – a real CPAN-like module repository!

In part 2 we will look at the developer guidelines I created for in-house modules in order to make them suitable for publishing in the repository.