Tome's Land of IT

IT Notes from the Powertoe – Tome Tanasovski

Category Archives: Uncategorized

PowerShell A 3-line Framework for Azure CLI

I haven’t had a lot of time to play in PowerShell as I’ve been spending all my time working on machine learning at work (I probably should blog about that stuff at some point).  However, the past two days I was fortunate to be a part of a financial services industry hackathon in cooperation with Microsoft around developing applications to help Doctors who work with patients with Autism.  Everyone wanted to do the fun data science work or work with the new Azure Video Indexer or the Kinect interfaces so I grabbed the dirt that no one wanted that I used to do day in and day out, i.e., Automate the build of the Azure environment.

I have personally been working with automating Azure since 2009.  In the PowerShell bible, I wrote a chapter on using the interfaces that were available at the time.  Quite laughable when you look at how it worked then.  I have experience automating infra with PowerShell, Terraform, and a few others that I really hate (ahem ARM – yuck and let’s never talk about DSC).  As it was a hackathon and I wanted to learn something new, I opted for Azure CLI.  I know many of the PowerShellers cringe at the idea of yet another command line that is not PS, but for me, I’ve never been a purist.  I looked at the interfaces and was impressed. In the spirit of the original PowerShell design, you can tap into what tool makes the most sense at the time.

After tinkering for a while there was some bad

What do you mean resources in a resource group have to have a name that is unique across all of Azure?

And there was some that was good

Wow it just works with a few parameters to the az command.

I realized that PowerShell provides the perfect tool to reduce the noise from the az commands.  I mean who really cares about tags or the id if they are never used later in a config file to connect to them.  What I wound up writing was a simple framework to do just that. I was able to simplify the creation of new services needed in Azure by any developer to exactly 3 lines of code:

Line 1: Verbose message explaining what is happening

This is really only for debugging and to make the code clear about what it’s doing:

Write-Verbose "Creating CosmoDB Account"

Line 2: Execute az command and provide output relevant to project configuration

The following executes an Azure CLI command. You should use $resourcegroup, $location, or $subscription as global script variables as needed.

$output = Invoke-AzureCLI "az cosmosdb create --resource-group $resourcegroup --name fsiautism$resourcegroup" |
        select name, type, documentEndpoint

Where the name must be unique across all of Azure, we are using fsiautism$resourcegroup as the name.

Finally, the select statement at the end selects the return properties that someone might need to consume the service. The script ulitmately creates JSON output that could be used later in a deploy script.

The Invoke-AzureCLI function simply deals with error handling and providing an appropriate error message if it fails.

Line 3: Add the output to the return object

Each notepropertyname gets converted into a root hash element that has your config. This name must be unique to any others already used in this script:

$returns |add-member -NotePropertyName CosmosDBAccount -NotePropertyValue $output

Resource Groups

One thing I really liked is that I can run a quick delete of a resource group and delete everything in it.  This means I can quickly start from a pristine environment whenever needed – say in a devops pipeline.

What’s Next?

Since we’re outputting only the things that are needed to connect to the service (maybe type doesn’t matter), we could easily take the json output that the script emits and pump it to another script that would place those names into the appropriate config so that everything can quickly point to a resource group.

The entire deployment script (inlcuding the magic function Invoke-AzureCLI that deals with all of the “stuff”) can be found in the FSI-Autism-Video hackathon project (currently in the develop branch, but should be in master soon).  And now thanks to this script, we are one step closer to allowing others to be able to get started hacking right away without trying to figure out the environment. It may not be the traditional Terraform or ARM, but honestly, I think this is way easier to bootstrap people into working on a project as it is very simple to craft and maintain and provides the ultimate flexibility.

Props to this blog post from Rob Boss for the technique around error validation I used in my Invoke-AzureCLI function.

Techstravaganza – PowerShell Track – Call to Speakers

Techstravaganza 2017 is in NY on April 28

You may or may not know, but I am a cofounder of Techstravaganza in NY.  I also manage the PowerShell track.  We are having our call to speakers for the event on 4/28.  That’s not a lot of time and I have no one to blame but myself.  However, we do have money to fly folks to NY for the day.  So please read, review, consider, and send us your abstracts.

Topics

While Techstravaganza is typically an IT pro conference, our PowerShell track does not always conform to that.  We seek level 300/400 content that is deep and interesting.  All topics and levels will be considered.  Please feel free to submit as many abstracts as you wish for consideration.  The more you submit, the better your chance that the group will decide on one of your topics.

What to submit:

Title: A catchy title for your talk
Abstract: A paragraph describing your talk that will be used on our website and track materials
Speaker Bio: A brief bio explaining who you are and any relevant twitter handles or blog links that you would like linked to from our website.

Please e-mail powershellnyc at gmail with the above

Timelines

Deadline 3/19
Decision 3/22

Expenses

For those travelling from outside of NYC, Techstravaganza typically reimburses travel expenses.  We guarantee up to 500 dollars, but in the past we have been able to recuperate all expenses for our speakers for the past 6 years .  The exact amount is dependent upon the origin of all speakers coming to the event.  Also, please note, these are reimbursements for airfare/hotel/taxi.  We do not cover additional expenses or provide an honorarium or stipend of any kind.

Sponsor Requests

If youare a sponsor interested in sponsoring the Techstravaganza, please also e-mail powershellnyc at gmail.com so that we may send you the list of sponsorship opportunities.

Announcing WinArduino – A PowerShell Module that helps you dev an Arduino from a Windows box

I recently starting taking the Arduino course on Coursera entitled “The Arduino Platform and C Programming” from the University of California, Irvine.  I was having a blast with the course, but I really wanted PowerShell interfaces while working on code.  For those that know me, they know that I like my standard interface while developing: conemu: with a split window for each thing I’m developing with vim in the top and powershell on the bottom or a putty session (also in conemu) with two tmux panes with vim on the top and bash on the bottom.

conemu_env.png

I wanted this same setup for Arduino while developing on Windows.  Hence I created the WinArduino project to provide the tools I needed to program the Arduino that can run in my bottom PowerShell pane.

Features

Here’s a list of the two major features in WinArduino:

Compile and Verify

Currently this requires ArduinoIDE to be installed on your system, but I may go deeper and eventually compile/upload directly without it.  For now, it suits my needs by providing a wrapper to the Arudino IDE debug commandline interfaces to compile or compile and upload to my Arduino over USB.

Compiling is done with Invoke-ArduinoVerify and compiling with upload is done with Invoke-ArduinoUpload.

Invoke-ArduinoUpload c:\sketches\Blink\Blink.ino

Serial Input/Output

In the examples folder on the project are some robust examples which include some Arduino sketches to show how you can use PowerShell to either read/debug from serial messages in your Arduino code or how you can use PowerShell as an interface to directly control your Arduino board.  The latter is pretty fun – the example simply lets you write a 0 or 1 to turn off or on an LED, but it really opens the door to a lot of really fun things you can control with PowerShell in the IOT-Arduino world.

Get Started Now!

So what are you waiting for?  Sign up for the coursera course, download the WinArduino module, buy yourself an Arduino kit, and start hacking the electrons in your circuits!

 

Powerbits 10.5 History Revisited hg & he

Not too long ago, I posted a powerbit for a function that I was using called hgrep.  This is the evolution of that function and how I use it ALL the time in Windows PowerShell.

Here are the two functions, hg (history grep) and he (history execute).  Note: hg is also used by mercurial so feel free to change the alias to suit your needs – you can get the gist with a license to use/modify it off of github.

function Get-HistoryGrep {
    param(
        [Parameter(Mandatory=$false, Position=0)]
        [string]$Regex
    )
    get-history |?{$_.commandline -match $regex}
}

function Invoke-History {
    param(
        [Parameter(Mandatory=$true, Position=0)]
        [int]$Id
    )
    get-history $Id |%{& ([scriptblock]::create($_.commandline))}
}

new-alias hg Get-HistoryGrep
new-alias he Invoke-History

Basically, the functions are used like this:

“Oh man, I need to re-execute that function again – it was an invoke-pester command, but it had a bunch of switches that I got wrong, and then finally got right”

PS C:\test> hg pester

  Id CommandLine
  -- -----------
   5 Invoke-Pester -TestName 'test1' -OutputXml out.xml
   6 Invoke-Pester -TestName 'test1' -OutputFile out.xml
   7 Invoke-Pester -TestName 'test1' -OutputFile out.xml -OutputFormat nunitxml

“Oh, that’s the one, number 7! Let me run that again”

PS C:\test> he 7

Now, you can reuse the above over and over without having to look it up. This is exactly how you code in Linux in the shell. The equivalent in Linux is this:

history |grep pester
!7

Unfortunately, we can’t use the “!” special character in PowerShell because it is reserved to inverse boolean values. However, the general workflow is the same.  If you develop, live, and probably one day die in the shell, then these two functions are essential to your daily life.

One final note, if you run over 4096 commands in the shell, by default “he” won’t work with the earliest commands.  You can modify the special variable, $MaximumHistoryCount, if you want to change this behavior

Look back at 2014 and look forward to 2015

A look back at 2014

2014 was the year of containers for me.  I spent a lot of time looking at Docker, Kubernetes, Mesos with Marathon and Aurora, and working (and using) tooling around the core Linux Kernel components that make these platforms possible.

shipping2

Additionally, this was the year that I fully embraced and understood Apache Zookeeper and played with etcd.  For me, Zookeeper becomes the foundation of nearly any distributed system I write.  It’s easy and it works.  Etcd has a place when zookeeper is overkill, but I have yet to use it in anything real.

Elasticsearch – I finally got dirty with elasticsearch.  I have a lot of positive things to say about it.  I’m interested in seeing how far the software can be taken.  Specifically to see whether the datastore (that can now be backed up) will become an actual data tier rather than just an efficient layer on top of a data tier.

In the Perl world, I learned Moose which make Perl actually usable in modern day programming.  It provides objects and types to Perl.

I was really happy to have the opportunity to implement ZeroMQ into something I was working on.  I am really excited by this library and I hope 2015 gives me a chance to write something about PowerShell and ZeroMQ.  There are very few platform and language agnostic libraries out there.  Additionally, the perfect abstraction with robust patterns and documentation make it a lot of fun to tinker with new communication topologies with just a few minor changes to code.  I have not been so inspired or zealously passionate about something like this since PowerShell 2.0 took my brain.

I played with AngularJS a bit.  It was fun.  However, every time I sit down to do UI work, I feel defeated.  It’s just something I’m not amazing at nor do I think I really want to be.  I’m glad to understand the framework and how it works, but I’ll save the meaty part of that work for others.

Pester Pester Pester!!! Test-driven development took over my PowerShell life (as well as all other languages).  My coding takes longer, but I have much more faith in it.  Pester is the greatest thing to happen to PowerShell from the community ever!  I was really happy to work with my internal corporate teams to build SDLC best-practices for PowerShell that involve Continous Integration (CI) and a Pester as the testing framework.

In 2014 I was introduced by my mentor to some time management techniques outlined in Eat that Frog.  Basically it’s about turning your life into Agile.  That sounds strange and the book doesn’t mention agile once – it is my interpretation of it.  It’s really about prioritizing daily and choosing the items that are most impactful to your company and yourself while deprioritizing everything else.  Additionally, I have adopted the more and more common practice of ignoring most e-mails and checking them less frequently throughout the day.  If it’s important, they will get back to you in a way that you cannot ignore.  Otherwise, it’s just getting in the way of the things you prioritized for the day.  If you start to follow this advice, I would add that you should also set up some alerts to ensure that certain people are never ignored.

An unhealthy obsession with gaming returned to my life in 2014.  However, I was successful in squashing it at the end of 2014 – well all of it except for the updates to candy crush I have to do when a new level comes out 🙂  Hopefully the squash will return some valuable time I need in order to blog a bit more and round myself in the wee hours of the night.

The 2015 Hitlist

New-Year-Baby-2

Golang

Use golang in an actual project.  I really like golang.  It feels like an interpretive language (PowerShell, Python, Perl), but it is compiled and has the potential to automagically make things more efficient as the engine matures.  I have played with golang a bit, but I want to find a project to use it with that will prove/disprove my initial thoughts about the language.

Openstack

Learn Openstack.  I’m sick of being in conversations where I cannot speak with authority about what open stack can and can’t do.  I need to understand all of the components and how they work.  This is pure lab time that I should have done last year.

Public cloud

Re-evaluate the cloud providers.  It’s been about two years since I last looked at AWS and Azure.  I’d like to get a handle on the current offerings and take a closer look at the Google compute engine stuff.

PowerShell Talks and Blogs

Put some new talks together about PowerShell 5.0 and the ZeroMQ library.  Perhaps finally blog or do a talk about the heuristic and deterministic algorithm implementations I have done with PowerShell.

Publish my unpublished article around running a PowerShell script as any user in AD without credentials 🙂 (it’s possible, but requires a configuration you will likely never want to do – but hey, from a research perspective, it’s fun to try).

Podcast

Revisit the cmdlet of the day podcast (no link because the storage is currently not working).  Of all of the things I have ever been involved with, this is the one that I get the most positive feedback from.  I have been thinking it would be fun to kick off the year giving 5-minute discussions about enterprise scripting best practices. There’s so much potential in the short-form podcast format for something highly technical.  I’d love to do this right and perhaps inspire others to pick up the torch in similar technologies that I could benefit from listening to.

The 2015 Watchlist

baby-with-glasses

 

PowerShell

PowerShell as a development language – I still firmly believe that PowerShell is one of the best languages ever written.  In my opinion it is a better interpreted development language than Python and Perl.  I would love to see it used the way it should be.  This is probably a losing battle as Microsoft’s focus is on making it feel like a development language strictly to get providers written for DSC, but I constantly hold my breath waiting for something more.  My hope is that the open sourcing of .NET along with the new language mode in 5.0 may open that door a bit more.  However, my face is slowly turning blue and I may not see the sweet release of air any time soon.  Additionally, I just don’t work on anything that would allow me to prove this outside of little pet projects here and there.  I suppose this is more of a crylist entry than a watchlist entry.

Microsoft open source

.NET being open sourced.  What does it mean?  What’s going to happen with it next?

Windows containers

Containers on Windows – What in the world does it all mean?  How will it manifest to IT shops, and how can I exploit it for the benefit of cheap, secure, and flexible compute where I work?

Checkpoint/Restore

In early 2013, I ran a successfull POC that leveraged CRIU to migrate an app including all of its state from one Linux server to another and have it start running again as if nothing happened.  Why is this not being exploited or am I missing projects that are leveraging it?  Either way, it’s still the most cutting-edge bit of magic out there.  I can’t wait to see where it goes.

Happy New Year!

%d bloggers like this: