Skip navigation

Category Archives: PowerShell

It’s a programming language! It’s a command shell! It’s PowerShell!

ISE_Cew and other ISE add-ons

A few weeks ago, I worked my way through a Lynda.com course on the Windows PowerShell ISE.

The current version of the ISE has an Add-On menu item that links directly to the PowerShell ISE Add-Ons page at microsoft.com.

One particular add-on was particularly interesting to me, since I’m trying to bring to bear what I know of proper software engineering practice to the PowerShell scripting I do.

This particular add-on, ISE_Cew, is a package that includes both Pester and Git functions. It’s just the thing if you’re looking to add Behavioral Testing (Pester) and version control (Git) to your PowerShell workflow.

testing PowerShell with Pester

I’ve been diving back in to – well, dipping my toe in the chilly waters of – PowerShell for some scripting here at my Data Processing job.

Several years ago, I learned the hard way (i.e., after writing a couple hundred lines of Ruby script) that although much of our processing automation was written without unit tests, that does NOT apply to any automation that *I* want to write. Not if I want to put it into production, that is.

I resisted unit testing and TDD for some time (Why? Well, that’s a story for another time), but I finally got testing religion last year with some Python scripting.

I could continue with the Python, but I think PowerShell is a better fit for our environment here.

Most modern programming languages have a choice of testing frameworks to choose from, but for PowerShell there’s only one that I know of – Pester.

Pester can be installed through NuGet or downloaded from GitHub.

I’m not going to repeat any Pester examples here – you can find plenty of “Getting Started” guides on the web. For example,

While looking for the Technet link, I found this post courtesy of Matt Wrock’s Hurry Up and Wait blog:

Why TDD for PowerShell? Or why pester? Or why unit test a “scripting” language?

Matt’s blog is subtitled “Tales from an Automation Engineer”, so his perspective on testing is a little different from the usual software testing guru. In particular, he points out that when it comes to infrastructure (and Data Processing, IMO), the things that are mocked / “stubbed out” in most software development environments are the things that we want to test:

Why TDD for PowerShell? Or why pester? Or why unit test a “scripting” language?

…

But infrastructure code is different

Ok. So far I don’t think anything in this post varies with infrastructure code. As far as I am concerned, these are pretty universal rules to testing. However, infrastructure code IS different…

If I mock the infrastructure, what’s left?

So when writing more traditional style software projects (whatever the hell that is but I don’t know what else to call it), we often try to mock or stub out external “ifrastructureish” systems. File systems, databases, network sockets – we have clever ways of faking these out and that’s a good thing. It allows us to focus on the code that actually needs testing.

However …if I mock away all of these layers, I may fall into the trap where I am not really testing my logic.

More integration tests

One way in which my testing habits have changed when dealing with infrastructure code is I am more willing to sacrifice unit tests for integration style tests…If I mock everything out I may just end up testing that I am calling the correct API endpoints with the expected parameters. This can be useful to some extent but can quickly start to smell like the tests just repeat the implementation.

Typically I like the testing pyramid approach of lots and lots of unit tests under a relatively thin layer of integration tests. I’ll fight to keep that structure but find that often the integration layer needs to be a bit thicker in the infrastructure domain. This may mean that coverage slips a bit at the unit level but some unit tests just don’t provide as much value and I’m gonna get more bang for my buck in integration tests.

Matt’s opinion accords with my intuition about my Data Processing environment. In the DP realm, the part of the script that can be tested without accessing the production environment (or at least a working model of the production environment) can be trivial. This is probably the main reason our existing production automation doesn’t have full testing coverage. (Well, that, and the fact that as far as I know there’s no testing framework for the automation software we use).

So I think my approach will be something like Matt’s – unit test where it’s useful and non-trivial, and more integration tests (a “thicker layer” as Matt says) to get full (or at least adequate) coverage.

PSFTP links

I’m trying to develop some scripts to handle data on some of the web servers we push data to at work.

I’m using PowerShell, because I can be fairly sure it will be available on the local hosts that we connect to the web servers from.

There is a “PSFTP” module that wraps .NET  calls in PowerShell commands, but it’s taken a few searches to find the best way to use it in our environment here.

The Technet article shows examples of using the PowerShell commands in the module, but the way the example sets up the initial FTP connection opens a dialog box (outside of PowerShell) for the user to enter the FTP password. This is ok for the example, but wouldn’t work for automated FTP applications.
Happily, I found a WindowsITPro article that shows how to use the PowerShell ConvertTo-SecureString
cmdlet to create the FTP credential using a stored password in plaintext (or from a configuration file), rather than having to be logged in to the server to enter the password and allow the script to continue:

Import-Module PSFTP

$FTPServer = 'ftp.host.com'
$FTPUsername = 'username'
$FTPPassword = 'password'
$FTPSecurePassword = ConvertTo-SecureString -String $FTPPassword -asPlainText -Force
$FTPCredential = New-Object System.Management.Automation.PSCredential($FTPUsername,$FTPSecurePassword)


Set-FTPConnection -Credentials $FTPCredential -Server $FTPServer -Session MySession -UsePassive
$Session = Get-FTPConnection -Session MySession

Get-FTPChildItem -Session $Session -Path /htdocs #-Recurse

Using the ConvertTo-SecureString cmdlet, I’ve been to create a connection and list the contents of the remote (FTP server) directory.

One thing I’ve had to look at with the Technet and WindowsITPro examples is the “-UsePassive” parameter on the Set-FTPConnection command. I’ve never had to be concerned with the difference between Active and Passive FTP modes before, but it appears that Passive mode has problems with the way our firewall is set up. I found a StackOverflow article about the difference between Active and Passive FTP, but I will probably need to keep consulting the article until I’ve really got a handle on it.

Finding connected Remote Desktop users

Finding Remote Desktop users connected to a Windows server using PowerShell

(without using the RemoteDesktopServices cmdlet – script uses WINSTA.EXE instead.

Where it all started – the Monad Manifesto

PowerShell originally started as a project called “Monad” within Microsoft.

The original Monad Manifesto[PDF] was written by Jeffrey Snover back in August 2002.

BTW, one of the major influences on Monad was a paper by John Ousterhaut:
Scripting: HigherLevel Programming for the 21st Century[PDF]

It’s interesting to read Snover’s original manifesto and see how much of the original vision made it into PowerShell (and how much didn’t).

(originally posted at edward.spurlock.cc)

Setting up my PowerShell profiles

There comes a time in every programmer’s life when s/he has to strike out on his/her own, writing new code (instead of typing in examples from books / websites). That time has come now for me with regards to PowerShell.

But first, I have to set up my working environment.

Here at work, we have a common (i.e., shared) network directory on our Production resource server. There were no PowerShell utilities in the directory (probably because I think I’m the first person to do anything serious with PowerShell here, with the possible exception of the IT guys – and they don’t use the Production resource server).

However, it occurred to me that that common directory (call it N:\common\utils, because that’s not its name) would be a good place to put modules meant to be shared.

How do I tell PowerShell to look for modules there, without having to specify this every time I start PowerShell?

For now, I just:

  1. created a PS subdirectory in N:\common\utils (PS for PowerShell, of course)
  2. Started PowerShell on my PC and created a profile file in the $profile directory (per Recipe 1.6 from the Windows PowerShell Cookbook):
    New-Item -type file -force $profile
  3. edited the profile file using Notepad.exe:
    notepad $profile
  4. and added a line to add the common directory to the PSModulePath environment variable:
    $env:PSModulePath += “;N:\common\utils\PS”
    (the leading semicolon is the profile path separator)
  5. exited notepad, saving $profile on the way out.

Now, whenever I start PowerShell, the $profile runs and adds the PS shared folder to the module search path.

To do the same thing (less step 1) for the Windows PowerShell ISE, I consulted Microsoft Technet article How to Use Profiles in Windows PowerShell ISE, which suggests wrapping the New-Item in step 2 in an if statement to prevent overwriting an existing profile, and using the ISE to edit the resulting profile file:

  1. (PS subdirectory already created in N:\common\utils)
  2. Started the PowerShell ISE and created a profile file in $profile:
    if (!(test-path $profile)) {New-Item -type file -path $profile -force}
  3. edited the profile file (using the ISE editor):
    psEdit $profile
  4. and added the same line to add the common directory to PSModulePath:
    $env:PSModulePath += “;N:\common\utils\PS”
  5. then closed the ISE editor tab, saving the ISE $profile file on the way out

Now I just have to figure out modules and module manifests…

(originally posted at edward.spurlock.cc)

Automating web browsing using PowerShell

A lot of our Production Quality Control (QC) operations where I work require checking that data has been uploaded to one of our websites, using either one of our internal tools, or our backdoor access to one of our customer-facing sites. This is all right when we’re checking a couple of customer jobs, but gets tedious VERY quickly for routine QC of dozens or hundreds of customer jobs.

A web app works well as a manual tool (“enter text to be searched for in this box, click Search, click the link for desired item in the list of items matching your search term…”), but our internal tools and customer-facing sites were never designed to be scripted.

For a while, when I was working with more cross-platform scripting languages, I was looking at Selenium. Selenium allows you to control popular web browsers from a number of programming languages, including Python, Ruby, and C# – but not directly from PowerShell. It would be possible to write my own PowerShell wrapper for C# to control Selenium, but I don’t have any experience extending PowerShell with C#, and since we’re not a C# shop, I think that would be very fragile from a long-term maintenance standpoint.

Anyway, unlike the typical Selenium application, our Production QC ops aren’t testing a single web app across multiple browsers. We’re searching for a multitude of data items, but we only have to find each one once, in a single web browser. A more robust solution would be to use something more native to PowerShell to control a single browser – which could even be Internet Explorer (perish the thought!).

I Googled “powershell web browser automation” and came up with a number of possibilities.

Web UI Automation with Windows PowerShell, is an MSDN article from 2008 and talks about using COM to control Internet Explorer, which is something I’ve dabbled in using VB Script. My first experiment with the method wasn’t successful, though, so I looked for troubleshooting info for COM in my handy copy of Windows PowerShell in Action. As it happens, the book illustrates COM with an example of “…a COM-based tool to help manage…browser windows,” so the book probably offers a more fertile field for further research.

A post on StackOverflow then led me to WatiN – Web Application Testing in .Net. WatiN allows control of Internet Explorer AND Firefox, so it might be even better than using COM.

(originally posted at edward.spurlock.cc)

Another PowerShell resource

Another resource I’ve been mining in the last week or so: PowerShell.org, an independent community for PowerShell users.

Something at PowerShell.org that you won’t find at every other PowerShell resource: the PowerScripting Podcast, currently on episode 289. As I get time, I’m paging back through the archives to find episodes that are of interest to me (a relatively new PowerShell user)

(originally posted at edward.spurlock.cc)

Three PowerShell resources

Microsoft’s Hey, Scripting Guy! blog

Microsoft TechNet Script Center

PowerShell Code Repository – PoshCode.org

(originally posted at edward.spurlock.cc)

Find subfolders that lack a given file(name)

Our processing automation at work creates a number of files during processing. One way we can tell when the automation hasn’t completed successfully is when the processed files directory has been created, but the files that are created at the end of processing are missing from the directory.

Here’s a PowerShell script fragment to identify subfolders (two levels down) that lack a given file (target.txt in this example) :

PS > Get-ChildItem |
>>   ForEach-Object {
>>     Set-Location $_
>>     Get-ChildItem |
>>       Where-Object {!(Test-Path $_\target.txt)}
>>     Set-Location ..
>> }
>>

And here’s a one-liner version of the above:

PS > ls | %{ cd $_ ; ls | ?{!(test-path $_\target.txt)} ; cd ..}

It’s not perfect – if it encounters a file (rather than a subdirectory) one level down, it attempts to Set-Location to that filename and throws an error (because you can’t Set-Location to a file, only a folder). However, it seems to find all the folders missing the target.txt file before it attempts to Set-Location to that file.

(originally posted at edward.spurlock.cc)