Tome's Land of IT

IT Notes from the Powertoe – Tome Tanasovski

Corporate Powershell Module Repository – Part 1 – Design and Infrastructure

Modules – Modules – Modules

If you’ve ever had a Powershell conversation with me the chances are high that you’ve heard my agenda around modules.  I think that the 2.0 module cmdlets were exactly what Powershell needed.  I come from the land of Perl where modules are an integral part of the community.  Well-formed reusable code is something Perl developers take a lot of pride in.  Unfortunately, the Powershell community has been very delinquent in picking up the Microsoft implementation of modules.  This is mainly due to the lack of a well-designed Internet repository that users can easily install and import from.  While this is a difficult task that we have yet to take on in the Powershell community, it is a much easier task to take on in a corporate environment.

Unfortunately, I can’t disclose the name of the company I work for, but I can tell you it is very large with about 100,000 users and plenty of IT folks to support it.  When I started there three months ago I was extremely surprised at how amazing their Perl repository and documentation was complete with wiki sites, in-house modules, and distribution of those modules through a globally distributed file system.  This was a fountain of information to drink from.  Then I went to the Powershell site: ugh – a well of cyanide!  The home page still referred to it as Monad!  The snapins that were installed to the distributed file system were versions behind, and the documentation really needed an overhaul that reflected the usage of both Powershell 1.0 and 2.0 together.

It was a daunting task, but I quickly volunteered to take it on.  Sweating nights and weekends I not only redesigned the documentation, but I put together a module repository that I am extremely proud of.  Here’s what I did:

Gathered Requirements

I first spoke with users of Powershell in the company to understand how it’s used, and to understand what ways a module repository could be helpful.  I came up with the following requirements:

  1. The repository had to support multiple versions of the same module
  2. Users who were using Powershell as a shell expected to have the most recent version of a module loaded when using Import-Module
  3. Users who were writing scripts needed the ability to specify a version number of the module to use in order to ensure that a new version of a module does not break their script.
  4. The repository needed to be available globally by all workstations and servers
  5. The repository required robust documentation for each module.

I had one additional requirement that came from me:  Use the built-in cmdlets for modules i.e. Import-Module and Get-Module

Leveraged Existing Infrastructure

The distributed file system they use is extremely custom and is AFS based.  While my design could use any CIFS share it was nice to have something with such strict control.  I can publish to a \dev workspace for testing.  Users only have access to DEV if they have their machines configured to use it via the registry.  After testing and development is complete I can lock the DEV instance of a module and publish the files to a read-only file system accessible by all users and servers in the company around the world.  The read-only piece is key.  If you wind up configuring your in-house module repository using my method I highly recommend making the repository read-only with only a few key accounts having access to publish.

For the remainder of this article we will call the published read-only share: \\powertoe\modulerepo\

Configure the Code Access Security Policy

This is the only piece that requires administrator privileges.  Because the distributed file system we use is already leveraged heavily by other .NET groups within the company this part was already in place in our desktop and server builds.  In order to allow .NET to trust a share so it can load .DLLs from it you need to modify the policy with caspol.exe.  Caspol.exe can be found in your .NET directory, and documentation about it’s usage can be viewed here.

Here’s a quick batch script that will determine the proper directory to run caspol.exe from, and then call caspol.exe with the proper parameters to configure our system to trust the \\powertoe\modulerepo share:

@echo off
setlocal
for /f %%a in ('dir %windir%\Microsoft.Net\Framework\v* /b') do @call :caspol %windir%\Microsoft.Net\Framework %%a
for /f %%a in ('dir %windir%\Microsoft.NET\Framework64\v* /b') do @call :caspol %windir%\Microsoft.Net\Framework64 %%a
endlocal
goto :EOF
:caspol
set sc_dotnetpath=%1
set sc_dotnetversion=%2
%sc_dotnetpath%\%sc_dotnetversion%\caspol -polchgprompt off
%sc_dotnetpath%\%sc_dotnetversion%\caspol -m -ag 1 -URL "file://///powertoe/modulerepo/*" FullTrust
%sc_dotnetpath%\%sc_dotnetversion%\caspol -polchgprompt on
goto :EOF

This step is only necessary if you are either using snapins that you will convert to modules or if your modules load dlls of any kind.

Design the Folder Structure

This was a big hurdle to overcome.  Because I needed to nail the versioning of modules for many different purposes I spent a lot of time toying with using the -version parameter of Import-Module.  What a dead end that was!  The -version parameter seems to have been an afterthought to versions: Documentation that doesn’t work and a backwards method of selecting the highest version of a module frustrated me to no end.  You can read a little more of this digression here.

After throwing away “Import-Module -version” I came up with an elegant solution to the problem.  I decided I could script the user’s $env:PSModulePath.  If you are not aware of how modules become available to the built-in cmdlets Import-Module and Get-Module you should read:

Get-Help about_modules

To summarize the relevant parts: the environment variable PSModulePath holds the file paths that the built-in cmdlets look in to find modules that are available.  For example if the PSModulepath is set to c:\modules and you have a directory in that path called ToeISE with a ToeISE.psm1 or ToeISE.psd1 file it will be an available module when you run:

Get-Module -ListAvailable

If it is in that list you can then load the module with the following:

Import-Module modulename

As I mentioned I did struggle with trying to use a folder structure like \\powertoe\modulerepo\modulename\version or even \\powertoe\modulerepo\modulename\version\modulename with the $env:PSModulePath pointing to \\powertoe\modulerepo, but nothing worked quite the way I needed it to.  In the end I used the following configuration:

\\powertoe\modulerepo\modulename\version\modulename

Design the Profile

To fix the PSModulePath to point to the latest version of every module in the repository I added the path of the latest module to $env:PSModulePath through a quick one-line script:

Get-ChildItem \\powertoe\modulerepo\|where{$_.psiscontainer}|foreach {Get-ChildItem $_.fullname|Where{$_.psiscontainer}|sort -Descending|select -First 1}|foreach {$env:PSModulePath += ";" + $_.fullname}

or for better readability:

Get-ChildItem \\powertoe\modulerepo\|where{$_.psiscontainer}|foreach {
    Get-ChildItem $_.fullname|Where{$_.psiscontainer}|sort -Descending|select -First 1
}|foreach {
    $env:PSModulePath += ";" + $_.fullname
}

In the end, however, I wrapped this one line in a script and published it to \\powertoe\modulerepo\modulerepo.ps1.  I documented that users should dot (.) source this script in their profile.  It’s a similar copy/paste operation, but it allows me to control the profile of our users if anything in the module repository ever needed to change.

. \\powertoe\modulerepo\modulerepo.ps1

There is one small problem with my method:  It may not scale very well.  As the module repository grows it is possible that the script that loads PSModulePath will be too cumbersome.  If that ever happens I can modify modulerepo.ps1 to include the full path for each module.  It would then be a part of the controlled distribution process of any module to update this script with the new version of the module uploaded.  Rather than introducing a manual point of failure I decided to tackle this problem if the module repository ever grew large enough where it would impact performance.  There is also the question of whether performance will ever be hindered by a PSModulePath that becomes too large.  Again, a problem I am resolved to tackling down the road if it ever manifests.  In the meantime, the above solution gives our users a nice way to use get-module -listavailable and import-module modulename to use the central repository.  This works for the latest version of any module, but it doesn’t handle how to specify a version to use.

Controlling Versions in Scripts

The solution is really a no-brainer.  Import-module supports the usage of paths with the -name parameter.  So simply users are instructed to use the full path to the .psd1 in their scripts:

Import-Module -name \\powertoe\modulerepo\PowerCLI\4.1.0\PowerCLI.psd1

Because we control what gets distributed to the module repository we can enforce standards e.g. Must have a psd1 file, must have inline help with specific fields filled out, must use proper namespaces, etc.  Perhaps I will share that bit of information at a later date with you, but I know that there’s something that will interest you much more than an article about best practices.  If you read the above example for Import-Module you’ll see that I’m using it on a .psd1 file for PowerCLI – Wait a minute – Does that exist?  Does that work?  Can anyone guess what I will writing about in a future article?

Recap

  • Create share and publish modules to \\server\share\modulename\version\modulename
  • run caspol to trust \\server\share
  • Use the one-liner in your $profile:
Get-ChildItem \\server\share\|where{$_.psiscontainer}|foreach {Get-ChildItem $_.fullname|Where{$_.psiscontainer}|sort -Descending|select -First 1}|foreach {$env:PSModulePath += ";" + $_.fullname}
  • Use the following to view available modules:
Get-Module -listavailable
  • Use the following to import modules from the repository:
Import-Module modulename
or
Import-Module -name \\server\share\modulename\version\modulename\modulename.psd1

Modules – Modules – Modules

They are a cool powerful way of sharing your well-formed code.  It is a great way to promote best-practices internally and enforce your policy on reusable code.  They are also an amazing utility that lets you quickly browse and install other people’s code.  I strongly suggest you start using them.  Next stop, let’s do this on the Internet people – a real CPAN-like module repository!

In part 2 we will look at the developer guidelines I created for in-house modules in order to make them suitable for publishing in the repository.

About these ads

12 responses to “Corporate Powershell Module Repository – Part 1 – Design and Infrastructure

  1. Pingback: Corporate Powershell Module Repository – Part 2 – Developer Guide « Tome's Land of IT

  2. i200908 August 23, 2010 at 8:46 pm

    Wow, very nice. thank you!

  3. Arposh September 15, 2010 at 1:53 pm

    I ran into a similar situation in my job where I needed to dynamically host a large number of MSI installs. Rather than have it scan the entire directory structure each time a person needed to access a dynamic list of MSI installs available, I created a task that scanned the structure once daily and piped it to a text file. The program would read the list of MSI installs and their location from the text file instead of completely dynamically.

    This has a caveat of not being updated instantaneously each time a user accesses it, but I found a few advantages to this method.

    1. Speed – Reading a text file is much quicker than scanning directory structures.
    2. Change Control
    A. Standard change – Changes were made at a set time of the day set by the scheduled task.
    B. Emergency change – If a change needed to be made immediately, the scheduled task could be run manually and have it update the text file.
    3. Backups/ChangeLog – When the scheduled task runs, it renames the old text file with the date that it was created. This creates a changelog for the share and provides an easy way to revert to a previous setup (provided you pause the scheduled task).

    -Arposh

  4. Michael September 22, 2010 at 6:32 pm

    Nice work – thanks for sharing! Have been building a PowerShell module myself, am up to the distribution stage – good to know what works elsewhere :). Am currently looking at implementing your methods in our environment.

    Looking forward to reading the developer guidelines – a lack of ‘best practice’ is something I’ve been struggling with. Much easier to see what others do than establish our own (that may not even be sensible!).

    Michael

  5. Matt Thompson March 26, 2011 at 4:18 am

    Great idea – it’s got me up early on a Saturday to test something similar! I’ve been thinking of deploying modules to individual machines, having some way of pushing them out but this seems neater. I’m not too keen on scripts have to call other scripts but I think I have no good reason for that!

    I’ve read your article a couple of times and believe I understand it well now – as such i wondered if this was a typo – should this:

    Import-Module -name \\powertoe\modulerepo\PowerCLI\4.1.0\PowerCLI.psd1

    be:

    Import-Module -name \\powertoe\modulerepo\PowerCLI\4.1.0\PowerCLI\PowerCLI.psd1

    Thanks,
    Matt

  6. Pingback: a Monkey's Den, Mike Chaliy's Personal Site: Introducing PsUrl and PsGet

  7. JJ August 8, 2011 at 10:14 pm

    Nice article. We do something similar in our organization, but with a few differences:

    – We don’t deploy multiple versions to the network share (no requirement to).
    – In the network share, we have scripts that a user can run to add/remove the current path to $Env:PSModulePath. It’s a little simpler since we don’t have to add a path for each module version.
    – Instead of using $profile to identify which repository to load modules from (dev|qa|prod), we have the luxury of interpreting the module path based on information about the server (domain membership).
    – For DLLs (modules or just managed code assemblies) we tend to copy those to a local temp directory and load from there. It avoids caspol and it also helps to mitigate file locks when deploying an updated assembly to the repository.

    • Tome August 10, 2011 at 10:04 am

      The reason we have to support multiple versions is because we don’t want anyone updating a module and then having it break thousands of scripts that depend upon it if an enhancement changes the behavior of a cmdlet.

      I have to post an update to this series. We have changed a lot of things recently. We wound up writing our own Get-Module and Import-Module functions, but I’ll save those details for a post.

      • Tome August 10, 2011 at 10:06 am

        Also, writing to temp is undesirable in our environment. We try to be as virtual friendly as possible. That means don’t use shared disk unless you absolutely need to.

  8. Peter Kriegel January 13, 2012 at 4:12 am

    Hello Tome!

    I have asked the Question:
    How to not deploy PowerShell Snapins or Modules and using it on every Client in a large enterprise?

    http://social.technet.microsoft.com/Forums/en-US/winserverpowershell/thread/2e7a4304-b1ea-451c-be14-b562e993638c

    Even Karl Mitschke put answers there.
    But no Answer was satisfying me!
    Your Article (walkthrough) is a epiphany to me!
    Thank you very much!
    Andrew Nurse from Microsoft has done PSGet cmdlets which use NuGet with PowerShell.
    So take a look at his page:
    NuGet + PowerShell = (also) Crazy Delicious

    http://vibrantcode.com/blog/2011/2/10/nuget-powershell-also-crazy-delicious.html

    My pain is, we are using Quest Active Roles Server here. So we are forced to use the Quest QAD cmdlets.
    So I can’t wait for your 3 Article in this serie: How to turn a snapin to a module and use it centralized ?
    THIS ROCKS MAN !!! Weeeehaaa…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 29 other followers

%d bloggers like this: