Tome's Land of IT

IT Notes from the Powertoe – Tome Tanasovski

Category Archives: Powershell

H2o – Machine Learning with PowerShell

This article will give you the basics to leverage H2o to create a neural net model in seconds and then show you how to consume that model all from Windows PowerShell.  We’ll assume you do NOT know what H2o is nor do you know anything about machine learning other than it exists.  It’s going to rely on the REST API as there is no module yet for H2o (HINT HINT – Great easy project if anyone wants to take it on).

But, machine learning is hard, isn’t it?  Don’t I need high-level mathematics and a degree?

A few years ago I took the flagship Stanford course on machine learning from Coursera and have a basic understanding of how the different algorithms and implementations work.  However, without day-to-day practice, the information slowly leaked from my brain.  Historically, the barrier to entry to implement machine learning has been a bit too much.  Even with libraries that implement the various algorithms, it simply required a ton of work to even get to the point where you could train a set of data.  Tuning that usually required graduate-level understanding of each algorithm and the math behind it.  If you’ve toyed in this space without getting serious, then you probably have a similar story.  All of that changed for me when starting to play with the python library SKLearn (won’t be discussing today) and the cross-platform open environment for analytics/machine learning called H2o.

Full disclosure, there is still some art to tweaking the algorithms and data to get an optimal data model, H2o makes it much easier.  For now, don’t’ worry about that.  I promise you’ll have a deep learning neural net training a data model in only a few minutes.

What is H2o?

You can read all about it on their website, but in my mind, H2o is an open platform to easily train data models, tweak them, and finally put them into production.  It comes with a set of libraries for python, java, and R, it has a REST API (which we will leverage today), and it even has a GUI if you want to stay out of code entirely.  Personally, I find the web GUI great, but use it mostly for examining data I’ve loaded or to quickly do something when I don’t feel like looking up the API/lib call.  When using it with PowerShell it becomes essential because the mapping of the API is directly against what you can do with Flow.

Isn’t Machine Learning All About Big Data?

No!  People have lost their minds when it comes to data.  Just because people have lots of data doesn’t mean you need that data.  Depending on the number of columns or properties your data has and how efficient you are storing data in your data sets (ideally, just the numbers), you can fit 5k rows on only ~200k of disk space.  A million rows is about 40 MB.  As long as you can fit your data set in a single server’s memory, you are outside of the realm of big data.  Whoever is telling you otherwise, is trying to sell you something.  Don’t get me wrong, you can run machine learning against large data sets, but most problems are not big data problems.

The Iris Data

Today, we will be using one of the most famous sets of open data, the Iris data.  You can read about the data and its properties here, but at a high level, there is a bunch of data about flower dimensions (pedal length/width & Sepal length/width) and a classification of 3 types of Iris species.  The goal is to use the data to create a model to predict the species of Iris based on the dimensions as input.

Quick start H2o

H2o is a complete running service that you will need to start on your computer with Java.

    1. Download java (if you don’t have it already)
    2. Download H2o and extract the h2o-version folder directly into c:\h2o
    3. Run H2o in PowerShell:
Java.exe -jar c:\h2o\h2o.jar

Alternatively, this will work as a background job if you want to keep everything in a single process:

Start-job -scriptblock {java.exe -jar c:\h2o\h2o.jar}

You can now browse to http://localhost:54321 to open the H2o Flow interface:

Step 1 Prep the Data

Prepping the data will proceed as follows:

  1. Import the Data
  2. Parse the Data
  3. Split the Data into Train and Test data

Import the Data

The API call for this is pretty straightforward. It allows you to pass a file path or URL to the ImportFiles function.

$url = "http://localhost:54321/3/{0}"
$iris_url = 'https://raw.githubusercontent.com/DarrenCook/h2o/bk/datasets/iris_wheader.csv'

"Import the IRIS data"
$importfiles_url = $url -f "ImportFiles"
$importfiles_body = "path=$iris_url"
$ret = Invoke-RestMethod $importfiles_url -Method POST -Body $importfiles_body

Parse the Data

This step is about having H2o interpret the data format and load it into a native H2o Data Frame. Oddly, this step is the most complex to navigate in the API even though with all of the defaults H2o does a perfect job of parsing the data. The challenge is that you must first call a function to autodetect the Parse settings, and then use the output from that function to the Parse function. This brings us to the biggest issue with the API as it lives today in v3. The challenge is that the Post data does not accept JSON, but all of the return data from API is JSON. The team promises to introduce JSON data natively in the near future, but until then you must post data in key/value form as specified by application/x-www-form-urlencoded. Unfortunately, there are no native libraries to do this Marshaling between data types so I wrote a quick helper function to do a single-level of parsing for you:

function ConvertTo-FormData {
	param(
		  [Parameter(ValueFromPipeline=$true)] [PSObject] $InputObject
		 )
	Begin {
		$output = ""
	}
	Process {
		foreach ($prop in $InputObject.psobject.properties |select -expandproperty name) {
			if ($InputObject.($prop).gettype().name -eq "Boolean") {
				if ($InputObject.($prop)) {
					$output += "$prop=true&"
				} else {
					$output += "$prop=false&"
				}
			} if ($InputObject.($prop).gettype().isarray) {
				# hacky for h2o collections
				if ($InputObject.($prop).name) {
					$output += "$prop=[{0}]&" -f ($InputObject.($prop).name -join ",")
				} else {
					$output += "$prop=[{0}]&" -f ($InputObject.($prop) -join ",")
				}
			}
			else {
				$output += "$prop=" + $InputObject.($prop) + "&"
			}
		}
	}
	End {
		$output.Remove($output.Length-1,1)
	}
}

With this function in hand, calling ParseSetup and Parse is very straightforward once you know the parameters.  The parameters for any command can usually be figured out by a combination of looking at the input schema in the documentation (located inside of Flow under “Help”) and by trying to use the widgets in flow to see what the parameters the function is going to need:

And here it is in PowerShell code:

"Run parse setup to find out how H2o thinks it should parse the data"
$parsesetup_url = $url -f "ParseSetup"
$parsesetup_body = 'source_frames=[{0}]' -f $iris_url
$ret = Invoke-RestMethod $parsesetup_url -Method Post -Body $parsesetup_body

"Parse the data into a real H2o dataframe"
$parse_url = $url -f "Parse"
$parse_body = $ret | select source_frames,parse_type,separator,number_columns,single_quotes,column_names,column_types,check_header,chunk_size |ConvertTo-FormData
$parse_body += "&destination_frame=iris&delete_on_done=true"
$ret = Invoke-RestMethod $parse_url -Method Post -Body $parse_body

In the above, the destination_frame is the name of the data as it will now live on H2o. Once the data is imported, you can see what it looks like in H2o Flow with this common name.

Async Polling

Some of the functions are run asynchronously. The Parse function is one of those. When H2o does something in this manner, it will provide you with a job id that you may poll. Unfortunately there are no callbacks available or web sockets to tap into. Here’s a helper function to do the polling:

function Wait-H2oJob {
	param(
		  [Parameter(Mandatory=$true, Position=0)]
		  [String] $JobPath
	)
	$notdone = $true
	while ($notdone) {
		$status = invoke-restmethod ("http://localhost:54321" + $JobPath) |select -ExpandProperty jobs |select -ExpandProperty status
		$status
		if ($status -eq "DONE") {
			$notdone = $false
		} else {
			sleep -Milliseconds 500
		}
	}
}

Finally, with this function in hand, you can see when the Parse function is done with the following:

wait-H2oJob $ret.key.URL

Caveat: the job key does not always return in exactly the same properties. You’ll see this in the future async calls we are about to make.

Split the Data

Even though there are only 150 rows in the dataset, we’re going to split the data so that we can train against a portion of the data and validate how well the model works against the other portion of the data. The following code will put 90% of the data into a new data frame named “train” and 10% of the data into a data frame named “test”:

"Split the data into an 90% training set and a 10% testing set"
$splitframe_url = $url -f "SplitFrame"
$splitframe_body = "dataset=iris&ratios=[.90,.1]&destination_frames=[train,test]"
$ret = invoke-restmethod $splitframe_url -Method Post -Body $splitframe_body
wait-H2oJob $ret.key.URL

Step 2 – Build the Model

It may not look like much, but this is all you need to implement a machine learning neural net in H2o:

$deeplearning_url = $url -f "ModelBuilders/deeplearning"
$deeplearning_body = 'training_frame=train&response_column=class&model_id=neural'
$ret = invoke-restmethod $deeplearning_url -Method Post -Body $deeplearning_body
wait-H2oJob $ret.job.key.URL

In the above, we are saying to use the data set named “train” and that we will be using the data in the columns to predict the column named “class”. Finally, we’ll be naming this data model “neural” to make it easy to access and view in H2o Flow.

Step 3 – Use the Model to Predict Against the Test Data

Finally, we’ll call the Predictions function against the Test data. We’ll then look at the MSE or mean squared error to see how well the model fits the additional data. The lower this number, the better.

$predict_url = $url -f "Predictions/models/neural/frames/test"
$ret = invoke-restmethod $predict_url -method POST -Body "predictions_frame=predicted_test_data"
$ret.model_metrics |select -expandproperty MSE

The output would look like this. The lower the MSE, the better the model is performing. Later when truly exploring the power of H2o, you will use this value to help you understand how well models are doing (and how poorly you may be overfitting your model to your data).

0.14980011539780046

Step 4 – Use the model for new data

Basically, by repeating the steps above of importing new data, you may predict against this data using the data model that now lives in H2o that we created named “neural”:

"Load something you want to predict"
@"
sepal_len, sepal_wid, petal_len, petal_wid
5.1,3.5,1.4,0.15
"@ |out-file -encoding ASCII c:\h2o\predict.csv
$importfiles_url = $url -f "ImportFiles"
$importfiles_body = "path=c:\h2o\predict.csv"
$ret = Invoke-RestMethod $importfiles_url -Method POST -Body $importfiles_body

"Run parse setup to find out how H2o thinks it should parse the data"
$parsesetup_url = $url -f "ParseSetup"
$parsesetup_body = 'source_frames=[{0}]' -f $ret.destination_frames[0]
$ret = Invoke-RestMethod $parsesetup_url -Method Post -Body $parsesetup_body

"Parse the data into a real H2o dataframe"
$parse_url = $url -f "Parse"
$parse_body = $ret | select source_frames,parse_type,separator,number_columns,single_quotes,column_names,column_types,check_header,chunk_size |ConvertTo-FormData
$parse_body += "&destination_frame=predictme&delete_on_done=true"
$ret = Invoke-RestMethod $parse_url -Method Post -Body $parse_body
Wait-H2oJob $ret.job.key.URL

"Let's leverage the data model we built earlier to predict against this new data frame"
$predict_url = $url -f "Predictions/models/neural/frames/predictme"
$ret = invoke-restmethod $predict_url -method POST -Body "predictions_frame=predictme_results"

Forunately, H2o also gives you access to the raw data so that you can inspect the results of the prediction. The following will spit out the data in the data frame named predictme_results where we are storing the prediction results:

$results_url = $url -f "Frames/predictme_results"
$ret = invoke-restmethod $results_url
$ret.frames.columns |select label, data

The output of the above looks something like the following.

label           data
-----           ----
predict         {0.0}
Iris-setosa     {0.9999987226382198}
Iris-versicolor {1.2773201404672E-06}
Iris-virginica  {4.16397577662733E-11}

This means that with 99.99999% accuracy, the model believes that the values given would make this Iris belong to the species of Iris-setosa. That’s some serious confidence that we have identified our flower!

The Code Completely

All of the above code is available in a single script along with the MIT license for you to reuse every bit of it as you see fit.

What’s next?  Learn H2o!

Learn H2o – The best book I’ve played with is the O’Reilly book, “Practical Machine Learning with H2o“.  All of the code is in Python, but it’s more important to learn the interfaces and the details of the various parameters available with each available algorithm in H2o.  As a PowerShell developer, you should be able to follow along.  If you can’t, it may be time to learn a little python too – it may be a substandard language to PowerShell, but it is leaps and bounds ahead of PowerShell in terms of ecosystem and libraries – it also has great visual code editing tooling with Jupyter and Zeppelin if you want to start getting serious with machine learning or analytics.

Just as important will be attempting to use H2o via Flow a little more.  The API is completely reflected by the way that Flow works.  Additionally, Flow’s help section has links to the API to learn the interfaces as well as the input/output schemas as you need to do more with H2o and PowerShell.

What’s next?  Grid Searches & AutoMLBuilder!

The best things to look at while reading about H2o are grid searches to do parameter tuning of data models and the AutoMLBuilder which will try all of the algorithms and do parameter tuning automatically for you.  However, both of these are more interesting when you get to Sparkling Water.

What’s Next?  Sparkling Water!

Spark was originally designed to pull in large sets of data from Hadoop into a distributed cluster of servers all in memory.  This sped up the processing you could do with Hadoop data.  Sparkling water is the library that connects H2o to Spark.  It also lets you run H2o on a Spark cluster.  If you want to take the technique learned in this article to a distributed platform with thousands of cores churning, sparkling water does this for you without any additional work.  It’s especially effective when trying to find optimal parameters for the algorithm arguments.  This is generally very compute intensive due to the nature of how grid searches and the AutoMLBuilder are basically brute force attempts to find optimal sets of parameters.  They naturally parallelize very nicely.

What’s Next? Productionize a Model Without Keeping H2o Running

Finally, H2o is a great tool for building models, but after the model is built you probably want something a little lower in weight.  The answer is to download the pojo/mojo.  In an upcoming non-powershell post, I’ll share my techniques for using the pojos outside of H2o to create very fast REST services with a GUI that can be called and scaled as needed.

Advertisements

Linux folks meet piped objects, Microsoft folks meet sed!

The world is a buzz around the announcement that Microsoft has open sourced PowerShell and released a working version of the language for Mac and Linux.

2000px-tux-svgpowershell_5-0_icon

Personally, I’ve been looking forward to this for a long time.  While I believe that PowerShell is a great language for development, I’m NOT excited about that aspect of it on Linux.  I think it will be a long time before the core dotnet becomes tuned to a point where it would perform comparably to Python as an interpreted language, but I do have hope.  No, the reason I have wanted this is because it is a super booster to bash!

Bash Booster – Objects in the pipe

I cannot tell you how many times I’ve tinkered in a Linux shell over the last few years and cursed the fact that I didn’t simply have objects.  Sure I can represent objects in CSVs or JSON, but to truly interact with them on a non-text level is frustrating in a shell after you’ve used PowerShell for so long.  In PowerShell, this is the way the world works:

Invoke-Webrequest http://blah.blah.blah |
ConvertFrom-Json|where {size -gt 100} |select name, size |export-csv output.csv

It’s all about data manipulation.  Grab data, filter, select properties to create new data sets, and output that data.  Additionally, you often process that data inline, for example:

...| ConvertFrom-Json| select @{name=SizeMB;expression={size/1MB}}| ...

And because everything is objects, you can easily write your own parsers, manipulators, or outputters that are duck-typed to work with the objects coming in.  It’s truly a game changer in the shell.

Native Linux Commands to PowerShell? – Hello sed!

For those unfamiliar with Linux or those who use Linux who are looking for the quickest way to convert your text into PowerShell objects, I’m here to tell you that sed is your friend.  Basically, the process is to turn your Linux output into CSV with sed and then use ConvertFrom-CSV to turn the CSV into PowerShell objects.  Of course, this assumes there isn’t a built-in way to switch the output of the command to CSV or JSON.  If there is a switch that exists to do so, it is always the best way to go.  For this article we’re talking about pure text output to objects.

We’re going to use the -r format of sed so that the regexes are more robust as far as what you can use in the regex.  We’re also going to use the form s/regex/replace/g which basically says (s)earch for regex and swap the contents with replace (g)lobally in the input string.

For this example, we’ll look at the output of ps -f:

10:06:04 PS /> ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
tome     22528  4516  0 09:42 pts/2    00:00:00 -bash
tome     22689 22528  0 09:45 pts/2    00:00:04 powershell
tome     22760 22689  0 10:06 pts/2    00:00:00 /bin/ps -f

As you can see there are some spaces or tabs between each record. We can easily parse the contents of this and replace those spaces with a comma. To do this with sed, we do the following:

10:08:12 PS /> ps -f |sed -r 's/\s+/,/g'
UID,PID,PPID,C,STIME,TTY,TIME,CMD
tome,22528,4516,0,09:42,pts/2,00:00:00,-bash
tome,22689,22528,0,09:45,pts/2,00:00:04,powershell
tome,22774,22689,0,10:08,pts/2,00:00:00,/bin/ps,-f

For now, let’s just ignore the fact that the -f is showing up with a comma in the last line. There is a fix to that which I will give an example of, but for now, let’s just convert this to PowerShell:

10:27:24 PS /> ps -f |sed -r 's/\s+/,/g' |ConvertFrom-Csv |select uid, cmd

UID  CMD
---  ---
tome -bash
tome powershell
tome /bin/ps


10:27:49 PS /> ps -f |sed -r 's/\s+/,/g' |ConvertFrom-Csv |Format-Table

UID  PID   PPID  C STIME TTY   TIME     CMD
---  ---   ----  - ----- ---   ----     ---
tome 22528 4516  0 09:42 pts/2 00:00:00 -bash
tome 22689 22528 0 09:45 pts/2 00:00:09 powershell
tome 23058 22689 0 10:28 pts/2 00:00:00 /bin/ps

If you really need to ensure that the spaces in cmd are preserved, there is a good stack overflow discussion about it here, but the shortcut outcome would be to do something like this:

10:29:55 PS /> ps -f |sed -r 's/\s+/XXX/8' |sed -r 's/\s+/,/g' |
sed -r 's/XXX/ /g' |ConvertFrom-Csv |Format-Table

UID  PID   PPID  C STIME TTY   TIME     CMD
---  ---   ----  - ----- ---   ----     ---
tome 22528 4516  0 09:42 pts/2 00:00:00 -bash
tome 22689 22528 0 09:45 pts/2 00:00:09 powershell
tome 23084 22689 0 10:30 pts/2 00:00:00 /bin/ps -f

One final note: if you are new to PowerShell from Linux, the format commands always go at the end of an object chain. They are designed to change the display output to the screen. Otherwise, you should not use them. They will change your objects and is likely not what you want to do if you have a pipe after the format command.

What about headerless commands such as ls -l?

If you look at the output of ls -l and the corresponding output of the CSV file from a sed, it looks like this:

10:35:05 PS /home/tome> ls -l
total 4
-rw-rw-r--  1 tome tome    0 Aug 19 10:33 file1
-rw-rw-r--  1 tome tome    0 Aug 19 10:34 file2
drwxrwxr-x 11 tome tome 4096 Aug 17 10:36 PowerShell
10:35:08 PS /home/tome> ls -l |sed -r 's/\s+/,/g'
total,4
-rw-rw-r--,1,tome,tome,0,Aug,19,10:33,file1
-rw-rw-r--,1,tome,tome,0,Aug,19,10:34,file2
drwxrwxr-x,11,tome,tome,4096,Aug,17,10:36,PowerShell

There are two problems with the above.  First, there is an extra line that has no relevant info.  Second, there is no header to tell PowerShell what the property names are for the objects.

Skipping a line with Select

Skipping the total line is easy using the skip argument to Select-Object:

10:35:10 PS /home/tome> ls -l |sed -r 's/\s+/,/g' |select -Skip 1
-rw-rw-r--,1,tome,tome,0,Aug,19,10:33,file1
-rw-rw-r--,1,tome,tome,0,Aug,19,10:34,file2
drwxrwxr-x,11,tome,tome,4096,Aug,17,10:36,PowerShell

Adding a custom header

ConvertFrom-CSV has a specific argument called header that allows you to supply a list that makes up what would be the header found in a CSV if one does not exist.  Here is how you can use it to convert the output of ls -l to actual PowerShell objects:

10:43:33 PS /home/tome> ls -l |sed -r 's/\s+/,/g' |select -Skip 1 |
ConvertFrom-Csv -Header @('mode','count','user','group','size','month','day','time','name') |Format-Table

mode       count user group size month day time  name
----       ----- ---- ----- ---- ----- --- ----  ----
-rw-rw-r-- 1     tome tome  0    Aug   19  10:33 file1
-rw-rw-r-- 1     tome tome  0    Aug   19  10:34 file2
drwxrwxr-x 11    tome tome  4096 Aug   17  10:36 PowerShell

 

Alternative to sed

Alternatively, you can use PowerShell in place of sed for replacing string contents. The pattern is generally like this:

10:43:55 PS /home/tome> ls -l |select -Skip 1 |%{$_ -replace '\s+', ','}
-rw-rw-r--,1,tome,tome,0,Aug,19,10:33,file1
-rw-rw-r--,1,tome,tome,0,Aug,19,10:34,file2
drwxrwxr-x,11,tome,tome,4096,Aug,17,10:36,PowerShell

Summary

PowerShell adds a lot more tools into your belt.  When it comes to data manipulation, the paradigm shift to objects over text is a game changer.  I’m personally really happy to have this flexibility and I can’t wait to take advantage of it!

 

Interprocess Communication (IPC) Between Asynchronous Events Triggered in C# to PowerShell

I played with Apache Zookeeper and PowerShell all weekend.  While I won’t dig into what I’ve been doing deeply in this article, I will tell you that the .NET interface requires a lot of asynch callbacks (event triggering).  Fortunately, I worked out a way to not only receive an event from C# (easy Bruce Payette taught us that years ago), but I also found a way to communicate variables into the runsapce of the action that gets executed when an event is triggered.

We’ll start with a simple C# class with an event to subscribe to.  In this case, I’m going to subscribe to the Changed event of my TestEvent.Watcher class.  This class has a single method which I can use to invoke the event with a string message attached to it.  This code should be invoked prior to running anything else in this post:

$code = @"
namespace TestEvent
{
    using System;

    public class Watcher
    {
        public delegate void ChangedEvent(object sender, WatcherEventArgs e);
        public event ChangedEvent Changed;

        public virtual void InvokeEvent(string message)
        {
            if (Changed != null) {
                Changed(this, new WatcherEventArgs(message));
            }
        }
    }
    public class WatcherEventArgs : EventArgs
    {
        public string message;
        public WatcherEventArgs(string message) {
            this.message = message;
        }
    }
}
"@
add-type -typedefinition $code

Here is the code to create the subscription and invoke the event. Register-ObjectEvent creates a job to run the event actions:

$watcher = new-object testevent.watcher

$job = Register-ObjectEvent -InputObject $watcher -EventName Changed -Action {
    $eventargs.message
}

$watcher.InvokeEvent('Triggering this message')

sleep 1 # just to ensure that the trigger happens
$job |receive-job

When we invoke, it looks like this:

PS C:\> .\test.ps1
Triggering this message

In Zookeeper, I needed to pass the current and valid connection object I had to the watcher so that when the event was triggered it would be sure to use the live object. Unfortunately, I didn’t see a way to pass arguments to the scriptblock, but I realized that I had full access to the $sender variable. In the above example,$sender is the testevent.watcher object I created and stored in the $watcher variable. Therefore, I figured I could just use add-member on $watcher to attach whatever objects I need to the action scriptblock.  These could then be accessed via $sender,

$watcher = new-object testevent.watcher
$watcher |add-member -NotePropertyName 'object' -NotePropertyValue 'value at watcher creation'

$job = Register-ObjectEvent -InputObject $watcher -EventName Changed -Action {
    $eventargs.message
    $sender.object
}

$watcher.InvokeEvent('Triggering this message')

sleep 1 # just to ensure that the trigger happens
$job |receive-job

The results of the above show that $sender.object indeed has the value I set when I registered the event.

PS C:\> .\test.ps1
Triggering this message
value at watcher creation

With that accomplished, I had to test whether or not the parent script could modify the $watcher object prior to the trigger and still set the value. This would enable me to pass live objects if they ever changed or needed to be updated by the parent script.

$watcher = new-object testevent.watcher
$watcher |add-member -NotePropertyName 'object' -NotePropertyValue 'value at watcher creation'

$job = Register-ObjectEvent -InputObject $watcher -EventName Changed -Action {
    $eventargs.message
    $sender.object
}

$watcher.object = 'value at watcher invokation'
$watcher.InvokeEvent('Triggering this message')

sleep 1 # just to ensure that the trigger happens
$job |receive-job

As you can see, success! The object is updated and passed at the time of the trigger rather than at the time of registration.

C:\> .\test.ps1
Triggering this message
value at watcher invokation

In case your interested, communication from the triggered job to the parent (the other half of IPC) is very easy. Simply use the messages/objects from the output of the scriptblock itself. In other words, your parent script will need to call receive-job and deal with the messages/output accordingly.

UPDATE

The above technique is sound and still useful for controlling messages to an event trigger.  It’s a very clean technique to use the $sender object.  However, I realized further in my testing and playing that it’s not always necessary.  Actions have access to GLOBAL and SCRIPT scope variable.  Not only can they access those variables on demand, but they can set them too.  This winds up being an even easier medium for transferring state because it’s actually just updating the live state.

Invoking the GetDiskSpaceInformation method of the IOfflineFilesCache COM interface with C# and PowerShell

This article could also be entitled, “Using an Inproc COM server in C# and PowerShell”.

Part I of this series shows how to invoke the GetDiskSpaceInformation method of the IOfflineFilesCache COM interface via C++. This was accomplished after failing miserably at trying to get it to work with C# and PowerShell. However, after I solved the problem in C++ I understood exactly how it worked and was able to ask better questions in Google to find out how to do the same in C#.

The challenge

First, to explain why it’s hard. When I first looked at the docs, I thought this would be easy. In PowerShell, a COM object can be invoked by instantiating a com object with new-object.  For example:

$app = new-object -comobject excel.application

Even if that didn’t work, I knew that often times C++ code could be Marshaled to and from the world of C# and managed code with a bit of tinkering. This is usually where you see pinvoke.net and code that leverages add-type.  However, in this case, the libraries do not exist in pinvoke.  Basically, because this is really COM, you cannot do this.  Also, because there are no tlb files associated, you cannot easily just use the interfaces like they are COM.

Just to be clear as to why this is the case:  This is a new interface.  It was plugged in by the Windows developers into the latest versions of Windows.  It’s implemented in COM so that other languages can get access to it.  However, it’s not fully at the point where it needs to be flushed into general use.  I expect that in the years to come, we’ll see these interfaces exposed with a tlb and eventually there may even be a PowerShell module that is used to manage the offline files cache directly.  However, if you want access before that day comes, you need to get crafty.

Finding GUIDs OLE/COM Object Viewer

The key to invoking this code from C# is to know the GUIDs that were used for both the CLSID and the interface we are trying to get access to.  This can be accomplished by running the OLE/COM Object Viewer.  For me, this was installed with my Visual Studio 2013 and can be found here: C:\Program Files (x86)\Windows Kits\8.1\bin\x86\oleview.exe.

Once in the GUI, you can browse to Object Classes->All Objects->Offline Files Cache Control

olecomviewer1

We’re looking for the GUID: 48C6BE7C-3871-43CC-B46F-1449A1BB2FF3

Next, if you double-click on that you’ll see the interfaces.  In our case, we want the IOfflineFilesCache interface.

olecomviewer2The GUID is 855D6203-7914-48B9-8D40-4C56F5ACFFC5

It should be noted that these GUIDs are static.  You do not need to run the COM viewer on your desktop if you are invoking the exact same interface that I am demonstrating.  The GUIDs are the same on ever computer.  However, this is here to show the generic steps that are needed to invoke the methods on one of these no-TLB COM interfaces.

CreateInstance

The first step is to create an instance of the CLSID using the GUID we found above.

Guid ID = new Guid("48C6BE7C-3871-43cc-B46F-1449A1BB2FF3");
Type idtype = Type.GetTypeFromCLSID(ID);
IOfflineFilesCache obj = (IOfflineFilesCache) Activator.CreateInstance(idtype, true);

It should be noted that the code we are using requires:

using System.Runtime.InteropServices;

Unfortunately, in the above bit of code, we are referencing the IOfflineFilesCache type, but it does not yet exist anywhere.  Therefore, we have to help C# know what this is by creating an interface with the ComImport attribute

Interface Attributes

I should note that everything I’m demonstrating is documented here.  However, it’s a bit cludgy to get through.  Also, there are a few key elements it leaves out.  Specifically, it neglects to inform you that when you create the interface, you must implement all of the methods that exist in the interface in exact order leading up to the method you care about using.  The best way to get the methods and the order they are implemented is to read the C++ header file.  This was one of the #include files I showed in part I of this article.  Specifically, you need to view cscobj.h.  A quick search of your hard drive should find it in an SDK folder.  Once you have read through this, you can create the interface in the proper order:

    [ComImport]
    [Guid("855D6203-7914-48B9-8D40-4C56F5ACFFC5"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
    interface IOfflineFilesCache
    {
        [PreserveSig()]
        int Synchronize();

        [PreserveSig()]
        int DeleteItems();

        [PreserveSig()]
        int DeleteItemsForUser();

        [PreserveSig()]
        int Pin();

        [PreserveSig()]
        int UnPin();

        [PreserveSig()]
        int GetEncryptionStatus();

        [PreserveSig()]
        int Encrypt();

        [PreserveSig()]
        int FindItem();

        [PreserveSig()]
        int FindItemEx();

        [PreserveSig()]
        int RenameItem();

        [PreserveSig()]
        int GetLocation();

        [PreserveSig()]
        int GetDiskSpaceInformation(ref ulong pcbVolumeTotal, ref ulong pcbLimit, ref ulong pcbUsed, ref ulong pcbUnpinnedLimit, ref ulong pcbUnpinnedUsed);

        // only need to go as far as the function you need, but rest here for completeness

        [PreserveSig()]
        int SetDiskSpaceLimits();

        [PreserveSig()]
        int ProcessAdminPinPolicy();

        [PreserveSig()]
        int GetSettingObject();

        [PreserveSig()]
        int EnumSettiingObjects();

        [PreserveSig()]
        int IsPathCacheable();
    }

You’ll notice that in the above, the GUID is the GUID we found when looking at the interface in the OLE/COM viewer.

Finished code, but let’s do it in PowerShell

So, now that we have the interface and an instance of it, the rest is easy.  The following final bit of code injects the C# into PowerShell via add-type.  I simply return the results as a collection and then convert it into an object in PowerShell, but you could just as easily modify the code to have the object returned directly from C#.

$code = @'
using System;
using System.Runtime.InteropServices;

public class offlinecache {
    public static ulong[] GetOfflineCache() {
        ulong pcbVolumeTotal=0, pcbLimit=0, pcbUsed=0, pcbUnpinnedLimit=0, pcbUnpinnedUsed=0;
        Guid ID = new Guid("48C6BE7C-3871-43cc-B46F-1449A1BB2FF3");
        Type idtype = Type.GetTypeFromCLSID(ID);
        IOfflineFilesCache obj = (IOfflineFilesCache) Activator.CreateInstance(idtype, true);
        int i = obj.GetDiskSpaceInformation(ref pcbVolumeTotal, ref pcbLimit, ref pcbUsed, ref pcbUnpinnedLimit, ref pcbUnpinnedUsed);
        ulong[] output = new ulong[5];
        output[0] = pcbVolumeTotal;
        output[1] = pcbLimit;
        output[2] = pcbUsed;
        output[3] = pcbUnpinnedLimit;
        output[4] = pcbUnpinnedUsed;
        return output;
    }

    [ComImport]
    [Guid("855D6203-7914-48B9-8D40-4C56F5ACFFC5"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
    interface IOfflineFilesCache
    {
        [PreserveSig()]
        int Synchronize();

        [PreserveSig()]
        int DeleteItems();

        [PreserveSig()]
        int DeleteItemsForUser();

        [PreserveSig()]
        int Pin();

        [PreserveSig()]
        int UnPin();

        [PreserveSig()]
        int GetEncryptionStatus();

        [PreserveSig()]
        int Encrypt();

        [PreserveSig()]
        int FindItem();

        [PreserveSig()]
        int FindItemEx();

        [PreserveSig()]
        int RenameItem();

        [PreserveSig()]
        int GetLocation();

        [PreserveSig()]
        int GetDiskSpaceInformation(ref ulong pcbVolumeTotal, ref ulong pcbLimit, ref ulong pcbUsed, ref ulong pcbUnpinnedLimit, ref ulong pcbUnpinnedUsed);

        // only need to go as far as the function you need, but rest here for completeness

        [PreserveSig()]
        int SetDiskSpaceLimits();

        [PreserveSig()]
        int ProcessAdminPinPolicy();

        [PreserveSig()]
        int GetSettingObject();

        [PreserveSig()]
        int EnumSettiingObjects();

        [PreserveSig()]
        int IsPathCacheable();
    }
}
'@

add-type -TypeDefinition $code

$output = ([offlinecache]::GetOfflineCache()) 
new-object psobject -Property ([ordered] @{
    VolumeTotal = $output[0]
    Limit = $output[1]
    Used = $output[2]
    UnpinnedLimit = $output[3]
    UnpinnedUsed = $output[4]
})

Here’s a trail of it running:

06:50:06 PS C:\Dropbox\scripts> .\GetOfflineCacheDiskInfo.ps1


VolumeTotal   : 469636214784
Limit         : 109076041728
Used          : 0
UnpinnedLimit : 109076041728
UnpinnedUsed  : 0

Announcing PshOdata – Cmdlets that make Odata easy

I’m sitting in the PowerShell summit and just left the stage where I presented how the Odata extensions work and shared my new project (contributors welcome) to bring PowerShell-driven Odata endpoints to the masses

To make it clear just how easy it is to use, here is the code required to create a web service around get-process and stop-process.

$class = New-PshOdataClass Process -PK ID -Properties 'Name','ID'
$class |Set-PshOdataMethod -verb get -cmdlet get-process -Params Name, ID -FilterParams Name
$class |Set-PshOdataMethod -verb delete -cmdlet stop-process -FilterParams ID
$class |New-PshOdataEndpoint

This will generate three files in a folder called odata that can be copied up to your IIS server. After an IISReset, the Process endpoint will be available via the following URLs:

I hope you enjoy, and please post issues on GitHub if you encounter any.

Github Project

You Know PowerShell is an Object-Oriented Language, Right?

I was recently reading an article that I’ll refrain from linking to for fear of a flame war. It infuriated me because it referred to PowerShell as a procedural/functional language that is antiquated and more like shell scripting.  I reflected on this for a while and I came to the realization that if I were to look at 99.99999% of the code out there I may think the same thing.  I also realized that there are reasons that PowerShell code appears procedural instead of object oriented.

This post will do a few things

  1. Show you how to do object-oriented programming with PowerShell using a traditional approach
  2. Show you the alternative (still object-based) so you can see why it’s generally used

A Class

Let’s describe a dog.

Properties

A dog has a name, color, and is some size.

$dogclass = new-object psobject -Property @{
   color = $null
   name = $null
   size = $null
}

A Constructor

While it may not be needed because you can instantiate the object without doing so (via psobject.copy()), object-oriented developers love their constructors. It’s also a good place to add some validation for your class. For example, in the below constructor we’ll restrict the size to either small, medium, or large.

function DogClass {
    param(
          [Parameter(Mandatory=$true)]
          [String]$name,
          [Parameter(Mandatory=$false)]
          [string]$color,
          [Parameter(Mandatory=$false)]
          [ValidateSet('Small','Medium','Large')]
          [String]$size
    )
    $dog = $DogClass.psobject.copy()
    $dog.name = $name
    $dog.color = $color
    $dog.size = $size
    $dog
}

Now you can start using your class

08:31:50 PS C:\> $lucy = DogClass -name Lucy -color brown -size large
08:45:07 PS C:\> $lucy

color                                       name                                        size
-----                                       ----                                        ----
brown                                       Lucy                                        large


A Method

A dog performs certain functions. For example, a dog is known to pee. We can modify our dog class with a method by using Add-Member. Also note that you can use the $this special variable to access the object that is invoking the method.

$DogClass |Add-Member -MemberType ScriptMethod -Name "Pee" -Value {
    "A warm refreshing pee trickles out of {0}" -f $this.name
}

With the method created, you can instantiate Lucy again from the modified class and access her new method.

08:50:50 PS C:\> $lucy = DogClass -name Lucy -color brown -size large
08:52:30 PS C:\> $lucy.Pee()
A warm refreshing pee trickles out of Lucy

Accessor Functions

Accessor functions help you protect the properties of an object. As in Perl and Python, there is no real protected or private property available to you. Therefore, you can use the same convention that these other languages use to supply an underscore prefix to any private methods or properties. It doesn’t actually prevent people from using it, but it’s a clear sign that they shouldn’t use it. For example, if we wanted to make the size property private, we would modify the class to look like this. Note: I’m adding the scriptmethod we created in one step using the -PassThru parameter of Add-Member.

$DogClass = new-object psobject -Property @{
   color = $null
   name = $null
   _size = $null
} |Add-Member -PassThru -MemberType ScriptMethod -Name "Pee" -Value {
    "A warm refreshing pee trickles out of {0}" -f $this.name
}

ScriptMethod

With the new _size property, it’s easy enough to modify our constructor to use the new property, but what if you want to control how people set or get the property of size. You can create an accessor function to do this. Basically, this is just a method that will set or return the data from _size.

$dogclass |Add-Member -PassThru -MemberType ScriptMethod -Name Size -Value {
    param(
        [Parameter(Mandatory=$false, Position=0)]
        $Size
    )
    if ($size) {
        $this._size = $size
    } else {
        $this._size
    }
}

Now, we can access the data in _size by using the accessor.

$lucy = DogClass -name Lucy -color brown -size large
"The dog is {0}" -f $lucy.Size()

We can also set the _size using the same function.

$lucy.Size('medium')

It’s important to note that I lost the property validator I have in the constructor. ScriptMethods break if you have an optional parameter that also has a validator. There are two ways to handle this. Either you add validation to the Size() method or you create two accessor functions, SetSize() and GetSize(). Both are acceptable and both are very easy to implement. Here’s an example of implementing your own validatior

ScriptProperty

A nice way to allow access to the pseudo private properties is to use the ScriptProperty member instead. This provides you with a way to perform a scripted action for a get, but it also allows you to do validation on a set. The difference is that it allows you to use the equal sign instead of a method to set the property.

Here’s what a ScriptProperty looks like. The first ScriptBlock is the get function and the second ScriptBlock is the set function.

$dogclass |Add-Member -MemberType ScriptProperty -name Size -Force -Value {
    $this._size
} {
    param(
        $Size
    )
    if (@('small','medium','large') -contains $size) {
        $this._size = $size
    } else {
        throw "This is not a valid size.  A size must be small, medium, or large"
    }
}

I’m using Force above to override the Size member I already created. In order to use the new property

In order to use this property, I can now do the following with my instantiated instance of Lucy.

PS C:\> $lucy.Size = 'blah'
Exception setting "Size": "This is not a valid size.  A size must be small, medium, or large"
At line:1 char:1
+ $lucy.Size = 'blah'
+ ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], SetValueInvocationException
    + FullyQualifiedErrorId : ScriptSetValueRuntimeException

PS C:\> $lucy.Size = 'medium'
PS C:\> $lucy.Size
medium

A nice thing about this technique is that your custom formatters will be able to see the property. In other words, it can be used in Select, Format-Table, Format-List, etc.

Class Inheritance

PowerShell doesn’t have real inheritance. However, the functionality of inheritance is very easily achieved. If you’re not familiar with inheritance, it’s pretty straightforward. Many times a class will have subclasses. For instance, a Scottish Terrier is a subclass of dog that is always small, black, and prefers to chase rats (rather than hunt like a beagle or keep your feet warm like a pug). Therefore, we can create a Scotty class that is based on the dog class, but has these properties set by default and has a method for hunting vermin.

Inherited Class

Because the definition of our class is simply an object, we can copy and modify it to create a new class that can be used later by other things.

$ScottyClass = $DogClass.psobject.copy()
$ScottyClass.color = 'black'
$ScottyClass._size = 'small'

One thing to note when creating an inherited class in PowerShell using this technique is that you need to recreate your constructors. This is probably the only unfortunate part of the pseudo-class inheritance that PowerShell offers. It basically forces override whether you want it or not on the constructor.

function New-Scotty {
    param(
          [Parameter(Mandatory=$true)]
          [String]$name
    )
    $dog = $ScottyClass.psobject.copy()
    $dog.name = $name
    $dog
}

Now we can use our new class:

$George = New-Scotty George

Overriding Methods

In the case of the Scotty class where we want to add a new method for catching rats, there’s nothing special that needs to be done. New methods are treated the same as on the base class. However, if you want to override a method that exists on the parent class, the technique is identical, but you’ll need to supply the force parameter to Add-Member. For example, if we want to override the Pee() method for the dog, we could do the following.

$ScottyClass|Add-Member -Force -MemberType ScriptMethod -Name "Pee" -Value {
    "The Scotty sniffs the tree.  A warm refreshing pee trickles out of {0}" -f $this.name
}

Note about PIA 2.0

Bruce Payette has a great set of code in PowerShell in Action 2.0 about extending PowerShell to wrap the OO-ability of the language in something that looks cleaner. The end result is you can define a class like this:

DogClass {
    property name 
    property size
    method Pee { "Oh yeah"}
}
$dog = new DogClass

While doing research for this post, I realized that I had never noticed this section of the book. We really need to make this a publicly available extension. It’s pretty simple to do, but I don’t want to paste his code due to copyright reasons. Actually, if you’re the kind of person who has been looking for OO in PowerShell and have never read Bruce’s book, you probably should because it will answer a whole lot more about PowerShell than just this for you.

Objects from New-Module

The PowerShell team added a shortcut way to do all of the above. Modules exist in their own closure. Because of this, they may contain their own scoped variables that can be stored within the module itself. There is a parameter on New-Module called -AsCustomObject that will convert your defined scriptblock module into a PowerShell object with note properties and script methods. Here’s an example of the Dog class using this technique.

function New-Dog {
    param(
        [Parameter(Mandatory=$true)]
        [string]$Name,
        [Parameter(Mandatory=$false)]
        [ValidateSet('small', 'medium','large', $null)]
        [string]$size,
        [Parameter(Mandatory=$false)]
        [string]$color
    )
    New-Module -Argumentlist @($name,$size,$color) -AsCustomObject {
        param(
            [Parameter(Mandatory=$true)]
            [string]$Name,
            [Parameter(Mandatory=$false)]
            [ValidateSet('small', 'medium','large', $null)]
            [string]$size,
            [Parameter(Mandatory=$false)]
            [string]$color
        )
        function Pee {
            "A warm refreshing pee trickles out of {0}" -f $Name
        }
        Export-ModuleMember -Function Pee -Variable Name, Size, Color
    }
}

If we look at the members that this creates, you’ll see it looks very similar to what we’ve been doing up until now.

$Lucy = New-Dog -Name Lucy
$Lucy |Get-Member

   TypeName: System.Management.Automation.PSCustomObject

Name        MemberType   Definition
----        ----------   ----------
Equals      Method       bool Equals(System.Object obj)
GetHashCode Method       int GetHashCode()
GetType     Method       type GetType()
ToString    Method       string ToString()
color       NoteProperty System.String color=
Name        NoteProperty System.String Name=blah
size        NoteProperty System.String size=
Pee         ScriptMethod System.Object Pee();

The PowerShell Way

So what we’ve been looking at is modern object oriented programming. However, it’s rarely seen in the wild with PowerShell even though it’s very simple to implement. Why is that exactly? Sure, there are plenty of beginners out there who weren’t even developers until now, but I believe it’s deeper than that. I personally know that this exists. However, I’ve only used it on one project.

The reality is that PowerShell lives and breaths objects through everything. Also, these objects are very flexible because they can be extended or modified on the fly. Functions can take objects as input and the type of Input provided by strongly typed objects can more easily be achieved with strongly type parameters. I see the param() block of a function as something that describes the properties of an object that I expect to see.

Now this is PowerShell to me. This is a paradigm shift in the way we do object-oriented programming. To me, when my peers ask me why I’m so PowerShell crazy, this is the answer. It looks like it’s functional, but it is far from that.

function New-Dog {
    param(
        [Parameter(Mandatory=$true)]
        [string]$Name,
        [Parameter(Mandatory=$false)]
        [ValidateSet('small', 'medium','large', $null)]
        [string]$size,
        [Parameter(Mandatory=$false)]
        [string]$color
    )
    New-Object psobject -property @{
        Name = $Name
        Size = $Size
        Color = $color
    }
}

function Invoke-Pee {
    param(
        [Parameter(Mandatory=$true, Position=0, ValueFromPipelineByPropertyName=$true)]
        [String]$Name
    )
    PROCESS {
        "A warm refreshing pee trickles out of {0}" -f $Name
    }
}

New-Dog -Name Lucy |Invoke-Pee
A warm refreshing pee trickles out of Lucy

Not only do I have a dog that can pee, but I also can use the same Invoke-Pee function for anything that has a name property. It can also invoke-pee on a number of dogs without having to create a loop.  If I really needed to be sure it was a dog, I could take the entire psobject as a parameter and validate that it had the properties that make it a dog.  Perhaps even write the Test-Dog function to do that.

Finally, I would personally wrap the above in a module named Dog and use a prefix more like New-Dog and Invoke-DogPee. Then it’s easy to import the module and inspect what methods are available using Get-Command -Module Dog. Again, this looks and smells like it’s functional, but it’s all objects and encapsulation, and it’s amazing!

Powerbits #10 – history |grep or hgrep in PowerShell

I’ve been working a bit more in Linux. Besides it inspiring me to write a lexical parser in python for PowerShell (don’t get your hopes up – it’s too much work, and I can’t do it), I keep getting little inspirations for cool projects. For example, someone needs to create tmux for PowerShell; I digress. This powerbit is from that inspiration. Recently, my laptop’s up and down arrows stopped working. Because of this, I have been relying on history |grep something in Linux so that I can quickly copy and paste that something back into the commandline. Context Swtich! Yesterday I was in PowerShell, and I got frustrated by the output of get-history (I really only want the commandline property) and the complexity it takes to filter for what I want using Where-Object.   As I was discussing this with a colleague, I said, let me just wrap this already and never worry about it again. So here it is, hgrep:

function hgrep {
    param(
        [Parameter(Mandatory=$false, Position=0)]
        [string]$Regex,
        [Parameter(Mandatory=$false)]
        [switch]$Full
    )
    $commands = get-history |?{$_.commandline -match $regex}
    if ($full) {
        $commands |ft *
    }
    else {
        foreach ($command in ($commands |select -ExpandProperty commandline)) {
            # This ensures that only the first line is shown of a multiline command
            if ($command -match '\r\n') {
                ($command -split '\r\n')[0] + " ..."
            }
            else {
                $command
            }
        }
    }
}

you can use it like this:

hgrep something

While, it’s bad practice to add the |ft command to the function, I find it useful so that I can use the -full switch to get everything I want in the format I want to see it:

11:17:22 PS D:\Dropbox\scripts> hgrep longfunction -full

   Id CommandLine                  ExecutionStatus StartExecutionTime              EndExecutionTime               
   -- -----------                  --------------- ------------------              ----------------               
   9 longfunction                        Completed 9/26/2013 11:00:00 AM           9/26/2013 11:14:41 AM

You can fork or clone this gist off of github here, or just run the following:

git clone https://gist.github.com/6715170.git

Learning a New Language – Perl, Python, and PowerShell – Part I – Executing a System Command

I have been doing development in some form or another for a long time. I learned Perl around 1997. I toyed with Basic and Pascal in high school and college, but it was nothing serious. Since that time, I have been fluent with C++, Perl, C#, and PowerShell while toying with languages like Javascript, Java, and a handful of others. Most recently I’ve tasked myself with learning Python for some new Linux scripts I have to write for work. When sitting down to learn a new language, I realize that there are topics that you must learn that go beyond the core syntax in order to feel comfortable tackling a development task (especially if you are more of an IT Pro than a developer). There are plenty of books and tutorials that can teach you the syntax of a language, but that’s really only the tip of the iceberg. I believe that these are very basic recipes that you need to learn about and store in your utility belt as quickly as possible in order to become efficient with a language.

Because, I’m seriously learning Python now, I thought I would document this as I go. And hey, since most people come to my blog to learn PowerShell and since I happen to know most of this stuff in Perl, I’ll explain how to do it in all three languages. I think it will also give people a fair assessment of the strengths, weaknesses, and overall quirks in the languages to help us have a more informed discussion about language in the wild. So without further ado, Part I of learning a new programming language.

Learning a New Programming Language – Perl, Python, and PowerShell – Part I – Executing a System Command

Generally the libraries that come with most languages these days can do everything you could ever dream of, but there are often occasions when the libraries fall short of a native command in Linux or Windows or there is a binary that does something better than you could ever code. For example, to rewrite robocopy.exe or rsync in any language is a waste of time. No offense intended to those who are writing libraries that duplicate their functionality – if they are robust and do what I need, I’ll be happy to use them 🙂

The following will break down the different ways to call external commands in each language while trying to satisfy the following three separate use cases that I generally have for system commands, i.e., execute and wait to get the exit code, execute and continue, execute and retrieve the output so that the text from STDOUT can be parsed or stored for later use. If you are not familiar with the term STDOUT it stands for standard output; this is the normal text output from a non-windowed command.

For the sake of this article, I am assuming that we are talking about external commands within a Windows operating system.

Perl

Perl has a few ways that you can interact with the system depending on what you expect to do with the external command. When it comes to simplicity, Perl gets it right.

Execute and wait to get the exit code – System

The following will run an ipconfig command and return an exit code to Perl:

$Exitcode = System("ipconfig");

If you want to suppress the output since you only care about the return code, you can redirect the command to NULL:

$Exitcode = System("iconfig >nul");

Execute and continue – Exec

Exec executes an external command and continues. You receive no indication about how the command ran.

Exec("robocopy d:\\folder1 d:\\folder2");

Again, if you want to suppress the output, you can redirect to NULL:

Exec("robocopy d:\\folder1 d:\\folder2 >nul");

Execute and retrieve the output – Backticks

If you want to capture the output of the command, you need to use backticks. For example, the following will populate the $ipinfo variable with the returned output from ipconfig. The print will display it to the screen.

$ipinfo = `ipconfig`;
Print $ipinfo;

PowerShell

PowerShell has many ways to launch system commands as well, but it’s mainly due to the fact that system commands can be run natively in PowerShell. Just type an executable and it will run. Because of this, you can spawn a background PowerShell job with start-job or execute a command on a remote computer with invoke-command. Mind you, I’m not showing all of the ways that you can execute a command, only the ways that I personally use over and over.

Execute and continue – Start-Process

Start-Process is very versatile. I personally like to use the alias start whenever I use this cmdlet. If you want to launch a command and not wait for the output, you can do the following:

start ipconfig

However, if you want to launch a command with command line arguments, you’ll need to use the ArgumentList parameter. This can be feel a bit awkward, but it’s not difficult. The following illustrates how to use robocopy with arguments in Start-Process:

start robocopy -ArgumentList 'D:\Folder1', 'D:\Folder2'

Execute and wait to get the exit code – -Wait & .Exitcode

You can wait for a command to finish by using the -Wait parameter

start robocopy -ArgumentList 'D:\Folder1', 'D:\Folder2' -Wait

There is also a passthru command that will give you the process object for the process you are starting. Here is my personal technique for getting the exitcode (yes, there is more than one way to get exit codes, this is just my preferred method):

(start ipconfig -wait -passthru).exitcode

Execute and retrieve the output

Generally if I need to parse the return text from a command I’ll just execute the command inline. For example to set the variable $ipconfig to the output of ipconfig.exe, you can do the following:

$ipconfig = ipconfig

In the above, $ipconfig is actually a collection of strings. Each line of the output is an element in the collection. If you expect it to return a single string, you’ll need to join the elements together with “`r`n”.

$ipconfig = $ipconfig -join "`r`n"

Finally, if you need to use spaces in your command, you’ll need to use the call operator (&). It won’t hurt anything to use the following for all of the commands you need from STDOUT, but just remember that it takes a collection of strings. For example, the following will work (note that d:\folder1 isn’t quoted, but it still works because it will attempt to convert anything it sees into a string):

& 'c:\path with spaces\robocopy.exe' d:\folder1 'd:\folder2'

However, this will fail:

& 'c:\path with spaces\robocopy.exe d:\folder1 d:\folder2'

To do the above, you would need to use Invoke-Expression, but I very rarely use that function as it is considered bad practice. The honest truth is that I care very little about injection attacks in most of my code. If I did care, I’m sure this article would be littered with it for discussion with every language.

For more information about all of the ways to call system commands from PowerShell, check out the following article.

Python

Python also has a few ways of calling external programs. According to the documentation, it appears that all use cases should now use the subprocess module. To launch an external command and return the exit code, you can simply use the call() method.

Execute and wait to get the exit code – Subprocess.Call()

Import subprocess
Exitcode = subprocess.call("ipconfig")

One thing to note, if you are using a function that exists in cmd.exe, the above won’t work. You must use the shell=True argument as well. For example, to use dir you would do the following:

Import subprocess
Exitcode = subprocess.call(["dir", "/ad", "c:\\"], shell=True)

You can do the above with a single string as well:

Exitcode = subprocess.call("dir /ad c:\\", shell=True)

Execute and continue – subprocess.Popen()

In order to execute without waiting for the exit code, you need to use the Popen() method of subprocess. Everything with this feels a bit more developery than it does with Perl and Python. However, the entire subprocess module is packed with a ton of things – useful things? I’m unsure as of yet, but here is what I know I need.

subprocess.Popen("ipconfig")

This technique shares STDOUT and partially STDIN. This gets really confusing when you do this:

t=subprocess.Popen("powershell")

The above is a fun party trick that will alternate between the PowerShell and Python prompts and interactive interpreters. It’s probably not a good idea to run interactive commands this way.

If you want to suppress the output from displaying, you can use the redirect to NULL like we saw earlier in this article when discussing Perl. However, there are some neat little tricks with the subprocess module that I think are worth learning about. You can set STDOUT to NULL when invoking Popen(). You can also set other streams such as STDERR this way. For example, the below will set STDOUT to NULL and then STDERR to STDOUT.

subprocess.Popen("ipconfig", stdout=(open(os.devnull, 'w')),stderr=subprocess.STDOUT)

Execute and retrieve the output – suprocess.PIPE

Similarly how we set stdout to null, we can also set stdout to equal PIPE so that we can later inspect the stdout property of the object returned by Popen(). The following illustrates this:

p = subprocess.Popen("ipconfig", stdout=subprocess.PIPE)
output=p.stdout.read()

Summary

Scripting languages by their nature are designed to be extremely versatile. There are sometimes just too many ways to do the same thing. This sort of sprawl can make things both extremely easy to use and extremely difficult to figure out at the same time. Because of this, it is really important to track the techniques that work for you and build up a library for your common use cases. Personally, I think the three use cases that I outlined in this article are more than sufficient: execute and wait to get the exit code, execute and continue, execute and retrieve the output. I think in the case of all three languages there are fair ways to accomplish these tasks.

The Python technique stands out because it is a bit cumbersome. To be fair, the Python subprocess module appears to be filled with tons o’ fun so it is understandable that it is a bit heavy to wield. To be further fair to Python, there are other ways of doing the above in Python using methods in modules and libraries that the documentation recommends you no longer use. Running commands in PowerShell at the command line and with the call (&) operator feels easy, but part of the reason for that is because PowerShell is intended to be a native shell to Windows. It almost seems unfair – like comparing how you run commands in ksh versus bash versus cmd. Actually, when you compare bash to PowerShell or even cmd to PowerShell, PowerShell has such quirks that it would probably lose that battle. As I think we’ll see in the upcoming articles, there are pros, cons, quirks, and angles to all 3 sides of this coin.

Get a list of metro apps and launch them in Windows 8 using PowerShell

I’ve been playing with Windows 8 store apps (metro apps) in C# and Xaml recently, and of course I thought it would be cool to launch these apps from within Windows PowerShell.  After some fruitless research trying to find a native way to do this in PowerShell or C#, I learned about a C++ interface called IApplicationActivationManager.

Fortunately, I was able to find some code on Stack Overflow that showed how to use this in C# (the post also listed the registry information I will describe in this article).  After some tweaking the code and the Add-Type syntax, I was able to expose the function I needed into PowerShell.  After that it was only a matter of wrapping the function into something that resembles PowerShell.

The code can be found on poshcode.

Get-MetroApp

This is a simple function that returns all of the metro apps installed on your machine.  Any filtering must be done with Where-Object.  Here’s a sample of what the cmdlet looks like when run:

23:48:22 PS D:\dropbox\scripts> Get-MetroApp |ft -AutoSize

EntryPoint                                                 ID                                                                      
----------                                                 --                                                                      
default.html                                               26720RandomSaladGamesLLC.HeartsDeluxe_kx24dqmazqk8j!App                 
Magic.UltimateTuner.App                                    26739TheMagicSoftware.UltimateTuner_hf01bqnspr91a!App                   
ContosoCookbook.App                                        4d1ddc26-6768-4a0d-97db-7cd72fc805c9_d6s28v2r30d6p!App                  
DataBindingTest.App                                        51da487c-5876-4ec7-9cad-fb5762b5032c_d6s28v2r30d6p!App                  
Tuner.App                                                  76f04ed9-49c3-434c-8d45-e392388e7e4d_d6s28v2r30d6p!App                  
BoxeeRemote.App                                            8c2b4686-67a7-46c2-aa34-5ac085571fed_d6s28v2r30d6p!App                  
Microsoft.Expression.DesignHost.Isolation.IsolationProcess App.a875b8f2e.a8a0c.a437b.aa93f.a4b3cf406091a_8wekyb3d8bbwe!Designer.App
DefaultBrowser.DefaultBrowserActivatableClass              DefaultBrowser_NOPUBLISHERID!Chrome                                     
HSW.Client.App                                             DiscoveryCommunicationsIn.HowStuffWorks_qa3esp0sj9xn6!App               
default.html                                               Microsoft.BingWeather_8wekyb3d8bbwe!App                                 
default.html                                               Microsoft.SkypeApp_kzf8qxf38zg5c!App                                    
MetroTwit.App                                              PixelTuckerPtyLtd.MetroTwit_5kbmb3e034y6r!App

The code behind this function is all about the registry. However, this is not an easy story to tell. The registry keys are not consistent. Basically, xaml apps are shows differently than HTML5 apps. Also, I’m still not entirely sure what DirectX apps will look like, but I assume that the Chrome browser is a C++ app so hopefully we’re covered there. Here is the logic behind what Get-MetroApp looks for in the registry:

The starting point for everything is in HKCU:\Software\Classes\ActivatableClasses\Package.  Each key is a separate app.  Each of these keys contains a Sever key and a key called ActivatableClassID.

The server key contains two keys.  The one of these that is not named BackgroundTransferHost.1 is the one that will have a property called AppUserModelID.  The value of this is what is needed in order to launch a metro app.  This is the ID property returned from Get-MetroApp.

Unfortunately, it’s not always obvious which app is which from this key so the Get-MetroApp also looks in the ActivatableClassID key for another key that has a CustomAttributes key with a property named AppObject.EntryPoint.   I found that the best key to try first was one called App if it existed, followed by app.wwa.  If neither of those is found, I iterate through each key looking for the AppObject.EntryPoint property.

In the case of xaml apps, the value of Appobject.Entrypoint appears to show the name of the app.  In html5 apps, it appears to only show default.html.  That is, unless there are other keys or sources out there that we (the community) have yet to discover.  In all of the apps that I had installed, I was able to tell which app was which by reading one or the other property returned by Get-MetroApp.

Start-MetroApp

You can ether pass an ID returned by Get-MetroApp

Start-MetroApp PixelTuckerPtyLtd.MetroTwit_5kbmb3e034y6r!Ap

or you can pipe one of the results from get-metroapp into Start-MetroApp.

Get-MetroApp |? entrypoint -match metrotwit |Start-MetroApp

I used the following to test that it was able to start every app i had installed:

Get-MetroApp |%{Start-MetroApp;sleep5}

The End?

I know this will come in handy to many. I have already added the module in my profile and have bound a few functions to allow me to quickly switch to specific metro apps very quickly. However, I’m not sure this is over yet. I’m curious if we will find better ways to grab the name of the app and link it with the ID. Please feel free to use the comments section of this page if you learn anything beyond what I have done.


Edit: I just learned about Get-AppxPackage in the Appx module. It appears to have a cleaner name, but it does not have the launch id required to launch the app. I’m sure it will be easy to correlate the packages to this name using some of the other keys. This will have to wait for another day, but it’s at least a lead to a better Get-MetroApp cmdlet.

Powerbits #9 – Get the directory of the calling script or get the current working directory of a session

This is one of my most used snippits of code. I generally put this at the beginning of every script. Every time I need it, I always say I need to post this as a powerbit so that I can find it more easily. Well, my wait is finally over, and you get the pleasant side effect of learning my most used trick.

The following bit of code will populate $currdir. If the script is run via a .ps1 file, it will set $currdir to be the directory where the .ps1 file lives. This will allow you to right click on a file and say “run with powershelll”, and have it use the files in that directory. The problem with this is that it will notwork if you are developing in ISE or in a powershell.exe host window. In that scenario, you generally want $currdir set to the directory your shell is in. This snippit will do just that, i.e., if it is not run via .ps1 file, it will load from $pwd (with some regex to strip out the psprovider nonesense).

$currdir = ''
if ($MyInvocation.MyCommand.Path) {
    $currdir = Split-Path $MyInvocation.MyCommand.Path
} else {
    $currdir = $pwd -replace '^\S+::',''
}

You can fork or clone this gist off of github here, or just run the following:

git clone https://gist.github.com/6715996.git
%d bloggers like this: