Skip navigation

Category Archives: Scripting

The Golden Age of Scripting –
focus on (back-end) coding, especially Python, Ruby, and Windows PowerShell

It’s been a long 3 1/2 weeks here.

The same day that I wrote my last post (Saturday, August 13), we discovered that one of our large web data servers was missing…everything.

This started a couple of weeks of time-consuming data restores. We were clocking overtime hours for the first time in months.

Now that the server is back to normal, we’re able to sit back and reassess processes here, which has led me to reassess my approach to scripting.

Over the past couple of months, I’ve been interested in testing PowerShell using Pester. However, after the past month, I think a better approach would be to use Ruby for applications that need testing (for example, scripts that delete data) and restrict PowerShell to building GUIs (since Ruby has been backward with regard to standalone GUIs) and read-only applications (for example, identifying incomplete data processing jobs)

I’m still planning to use version control (Git) with PowerShell. I’m not CRAZY.

 

 

 

A few weeks ago, I worked my way through a Lynda.com course on the Windows PowerShell ISE.

The current version of the ISE has an Add-On menu item that links directly to the PowerShell ISE Add-Ons page at microsoft.com.

One particular add-on was particularly interesting to me, since I’m trying to bring to bear what I know of proper software engineering practice to the PowerShell scripting I do.

This particular add-on, ISE_Cew, is a package that includes both Pester and Git functions. It’s just the thing if you’re looking to add Behavioral Testing (Pester) and version control (Git) to your PowerShell workflow.

I’ve been diving back in to – well, dipping my toe in the chilly waters of – PowerShell for some scripting here at my Data Processing job.

Several years ago, I learned the hard way (i.e., after writing a couple hundred lines of Ruby script) that although much of our processing automation was written without unit tests, that does NOT apply to any automation that *I* want to write. Not if I want to put it into production, that is.

I resisted unit testing and TDD for some time (Why? Well, that’s a story for another time), but I finally got testing religion last year with some Python scripting.

I could continue with the Python, but I think PowerShell is a better fit for our environment here.

Most modern programming languages have a choice of testing frameworks to choose from, but for PowerShell there’s only one that I know of – Pester.

Pester can be installed through NuGet or downloaded from GitHub.

I’m not going to repeat any Pester examples here – you can find plenty of “Getting Started” guides on the web. For example,

While looking for the Technet link, I found this post courtesy of Matt Wrock’s Hurry Up and Wait blog:

Why TDD for PowerShell? Or why pester? Or why unit test a “scripting” language?

Matt’s blog is subtitled “Tales from an Automation Engineer”, so his perspective on testing is a little different from the usual software testing guru. In particular, he points out that when it comes to infrastructure (and Data Processing, IMO), the things that are mocked / “stubbed out” in most software development environments are the things that we want to test:

Why TDD for PowerShell? Or why pester? Or why unit test a “scripting” language?

But infrastructure code is different

Ok. So far I don’t think anything in this post varies with infrastructure code. As far as I am concerned, these are pretty universal rules to testing. However, infrastructure code IS different…

If I mock the infrastructure, what’s left?

So when writing more traditional style software projects (whatever the hell that is but I don’t know what else to call it), we often try to mock or stub out external “ifrastructureish” systems. File systems, databases, network sockets – we have clever ways of faking these out and that’s a good thing. It allows us to focus on the code that actually needs testing.

However …if I mock away all of these layers, I may fall into the trap where I am not really testing my logic.

More integration tests

One way in which my testing habits have changed when dealing with infrastructure code is I am more willing to sacrifice unit tests for integration style tests…If I mock everything out I may just end up testing that I am calling the correct API endpoints with the expected parameters. This can be useful to some extent but can quickly start to smell like the tests just repeat the implementation.

Typically I like the testing pyramid approach of lots and lots of unit tests under a relatively thin layer of integration tests. I’ll fight to keep that structure but find that often the integration layer needs to be a bit thicker in the infrastructure domain. This may mean that coverage slips a bit at the unit level but some unit tests just don’t provide as much value and I’m gonna get more bang for my buck in integration tests.

Matt’s opinion accords with my intuition about my Data Processing environment. In the DP realm, the part of the script that can be tested without accessing the production environment (or at least a working model of the production environment) can be trivial. This is probably the main reason our existing production automation doesn’t have full testing coverage. (Well, that, and the fact that as far as I know there’s no testing framework for the automation software we use).

So I think my approach will be something like Matt’s – unit test where it’s useful and non-trivial, and more integration tests (a “thicker layer” as Matt says) to get full (or at least adequate) coverage.

I’m trying to develop some scripts to handle data on some of the web servers we push data to at work.

I’m using PowerShell, because I can be fairly sure it will be available on the local hosts that we connect to the web servers from.

There is a “PSFTP” module that wraps .NET  calls in PowerShell commands, but it’s taken a few searches to find the best way to use it in our environment here.

The Technet article shows examples of using the PowerShell commands in the module, but the way the example sets up the initial FTP connection opens a dialog box (outside of PowerShell) for the user to enter the FTP password. This is ok for the example, but wouldn’t work for automated FTP applications.
Happily, I found a WindowsITPro article that shows how to use the PowerShell ConvertTo-SecureString
cmdlet to create the FTP credential using a stored password in plaintext (or from a configuration file), rather than having to be logged in to the server to enter the password and allow the script to continue:

Import-Module PSFTP

$FTPServer = 'ftp.host.com'
$FTPUsername = 'username'
$FTPPassword = 'password'
$FTPSecurePassword = ConvertTo-SecureString -String $FTPPassword -asPlainText -Force
$FTPCredential = New-Object System.Management.Automation.PSCredential($FTPUsername,$FTPSecurePassword)


Set-FTPConnection -Credentials $FTPCredential -Server $FTPServer -Session MySession -UsePassive
$Session = Get-FTPConnection -Session MySession

Get-FTPChildItem -Session $Session -Path /htdocs #-Recurse

Using the ConvertTo-SecureString cmdlet, I’ve been to create a connection and list the contents of the remote (FTP server) directory.

One thing I’ve had to look at with the Technet and WindowsITPro examples is the “-UsePassive” parameter on the Set-FTPConnection command. I’ve never had to be concerned with the difference between Active and Passive FTP modes before, but it appears that Passive mode has problems with the way our firewall is set up. I found a StackOverflow article about the difference between Active and Passive FTP, but I will probably need to keep consulting the article until I’ve really got a handle on it.

I’ve been studying Python for several years, off and on, but I’m only now getting traction on applying it at work.

One of the things I’m struggling with right now is how best to arrange the directory structure when using object-oriented design where some objects inherit from others, when I’m doing the development on my desktop machine but plan to move the project to a common network directory.

I found a post on Stack Overflow that referenced a post by Jean-Paul Calderone: Filesystem structure of a Python project – the Stack Overflow post got over 100 upvotes, which is a pretty good indication that Calderone’s structure is a good and reasonable one.

Maybe what I want is a relative import, as in this Stack Overflow post – but I can’t seem to make it work with my code just yet. Grrrr…

I thought I bookmarked this a long time ago – apparently I didn’t.

O’Reilly’s Test-Driven Development with Python (by Harry Percival) is still available to read online at the O’Reilly Atlas site (formerly O’Reilly’s Chimera Labs, if I remember correctly and if the URL is any guide).

Percival’s book was made available to read for free courtesy of O’Reilly’s Open Source Convention (a.k.a. OSCON). Since my budget is currently limited, I’m very happy for the opportunity to read this book without having to buy it (or keep checking it out of the library).

Finding Remote Desktop users connected to a Windows server using PowerShell

(without using the RemoteDesktopServices cmdlet – script uses WINSTA.EXE instead.

I just discovered that RubyConf 2015 will be just down the road in San Antonio, a short drive away. (see what you miss when you don’t attend user group meetings?!)

A short article about the programming learning process via Viking Code School’s blog:

Why Learning to code is So Damn Hard

I’ve just started reading it, but already I like this quote from Quincy Larson at Free Code Camp:

… was convinced that the seemingly normal programmers I ran into were actually sociopaths who had experienced, then repressed, the trauma of learning to code.

PowerShell originally started as a project called “Monad” within Microsoft.

The original Monad Manifesto[PDF] was written by Jeffrey Snover back in August 2002.

BTW, one of the major influences on Monad was a paper by John Ousterhaut:
Scripting: HigherLevel Programming for the 21st Century[PDF]

It’s interesting to read Snover’s original manifesto and see how much of the original vision made it into PowerShell (and how much didn’t).

(originally posted at edward.spurlock.cc)