PowerShell Multithreading: A Deep Dive (2023)

At some point, most people will run into a problem that a simple PowerShell script is too slow to solve. This could be collecting data from many computers on your network or creating many new users in Active Directory at the same time. These are great examples of how using more processing power would make your code run faster. Let's see how to solve this with PowerShell multithreading!

No reader? Check out this related tutorial video!

Don't you see the video? Make sure your ad blocker is turned off.

The default PowerShell session is single-threaded. Executes a command and when finished, moves on to the next command. This is nice because it keeps everything repeatable and doesn't consume a lot of resources. But what if the actions performed don't depend on each other and you have CPU resources to spare? In that case, it's time to think about multi-threading.

In this article, you'll learn how to understand and use different PowerShell multithreading techniques to process multiple data streams at the same time, but manage them from the same console.

Understanding PowerShell multithreading

Multithreading is a way of executing more than one command at the same time. Although PowerShell typically uses a single thread, there are many ways to use more than one thread to parallelize your code.

The main benefit of multithreading is to reduce code execution time. This time reduction comes at the expense of a higher processing power requirement. With multi-threading, many actions are performed simultaneously, which requires more system resources.

For example, what if you want to create a new user in Active Directory? In this example, there is nothing about multi-threading, since only one statement is executed. That all changes when you want to create 1000 new users.

Without multithreading it wouldNew ADUuserCommand 1000 times to create all users. It can take three seconds to create a new user. It would take almost an hour to create all 1000 users. Instead of using one thread for 1000 commands, you can use 100 threads, each executing ten commands. Instead of about 50 minutes, it now takes less than a minute!

Please note that you will not see the perfect size. The process of enabling and disabling elements in code will take some time. With a single thread, PowerShell needs to run the code and that's it. With multi-threading, the original thread used to run your console is used to manage the other threads. At some point, that original thread will time out, just all other threads will stay on the thread.

PowerShell multithreading prerequisites

In this article, you'll learn how PowerShell multithreading works in practice. If you want to participate, here are some things you'll need and some details about the environment used.

• ventanaPowerShell version3 or later: Unless specifically noted, all code shown works on Windows PowerShell version 3 or later. Windows PowerShell version 5.1 is used for the examples.
• Spare CPU and memory: You need at least a little more CPU and memory to run side by side with PowerShell. If you don't have them available, you may not see a performance benefit.

Priority #1: Fix your code!

Before you speed up your PowerShell multi-threaded scripts, you need to do some preparatory work. First you need to optimize your code.

While you can add more features to your code to make it run faster, multithreading introduces a lot of additional complexity. If you can speed up your code before multi-threading, do it first.

Identify bottlenecks

One of the first steps in parallelizing your code is figuring out what's slowing you down. Code can be slow due to bad logic or extra loops where you can do some modifications to run faster before multithreading.

An example of a common way to speed up your code is to change the filtering to the left. When you're interacting with a data set, any filtering you want to enable to reduce the amount of data should be done as soon as possible. Below is a code example to determine the amount of CPU used by thesvchostProcess.

The following example reads all running processes and then filters out a single process (svchost🇧🇷 It then selects the CPU property and makes sure the value is not null.

PS51>get process🇺🇸 Where-Object {$_.ProcessName -eq 'svchost'} | Object Selection CPU |where object{$_.CPU-ne $null}

Compare the above code with the following example. Below is another sample code that has the same output but is organized differently. Note that the following code is simpler and moves all possible logic to the left side of the vertical bar symbol. This preventsget processreturn processes that do not interest you.

PS51> Get-Process -Name 'svchost' | Where-Object {$_.CPU -ne $null} | select-cpu-object

Below is the time difference to operate the top two lines. While the 117ms difference is negligible if you run this code just once, it will increase if you run it thousands of times.

Using thread-safe code

Then make sure your code is "thread safe". The term "thread-safe" refers to the fact that thread-executing code can execute the same code at the same time and not cause a conflict.

For example, writing to the same file on two different threads is not thread-safe because it doesn't know what to add to the file first. Whereas two threads reading from a file is thread safe as the file is not modified. Both threads get the same result.

The problem with multithreaded PowerShell code that is not thread-safe is that it can get inconsistent results. Sometimes it can work fine because the threads have the right time to not cause conflicts. In other cases, conflicts arise that make troubleshooting difficult due to inconsistent errors.

If you're only running two or three jobs at a time, they might end up lining up exactly where they're writing to the file at different times. So if you scale the code to 20 or 30 jobs, the probability that at least two of the jobs won't try to write at the same time decreases.

Parallel execution with PSJobs

One of the easiest ways to run multiple threads in a script is withPSEmpleos🇧🇷 Cmdlets are integrated into PSJobsMicrosoft.PowerShell.CoreModule. EastMicrosoft.PowerShell.CoreThe module is included in all versions of PowerShell since version 3. The commands in this module allow you to run code in the background while keeping other code running in the foreground. You can see all the available commands below.

PS51 > Get-Command *-Job

track your work

All PSJobs are in one ofeleven states🇧🇷 With these states, PowerShell manages the jobs.

Below is a list of the most common states in which a job can be found.

• cerrado– The job is complete and the output data can be recovered or the job can be deleted.
• Operation– The job is currently running and cannot be deleted without interrupting the job. The output is also not recoverable.
• covered– The task is still running, but the host will be prompted for information before continuing.
• It has failed– A completion error occurred during task execution.

To get the status of a started job, use theget a jobDomain. This command retrieves all the attributes of your jobs.

Below is the result of a job where you can see the statusCerrado.The following example runs the codestart sleeping5in a job withStart-JobDomain. The status of this job is then returned withget a jobDomain.

PS51> Start-Job -Scriptblock {Start-Suspend 5}PS51> Get-Job

When the order status returnscerrado, it means that the code in the script block will be executed and exited. You can also see thatI have more datathe property isINCORRECT🇧🇷 This means that no output can be provided once the job is complete.

Below is an example of some of the other statuses used to describe job titles. you can see fromdomainSpine, which may have caused some of these jobs to fail, such as B. Trying to sleepa B CSeconds resulted in a failed job.

See Also

How to run PowerShell scripts for OrchestratorRun PowerShell scripts in parallel

create new jobs

As you saw above, theStart-JobThe command allows you to create a new job that starts executing the code in the job. When you create a job, you provide a script block that is used for the job. PSJob then creates a job with a unique ID number and starts running the job.

The main benefit here is that it takes less time to run.Start-JobCommand to execute the script block we are using. You can see in the image below that the command didn't take five seconds, it only took 0.15 seconds to start the job.

The reason the same code can run in a fraction of the time is because it runs in the background as PSJob. It took 0.15 seconds to make the code run in the background instead of running in the foreground and actually sleep for five seconds.

Get job output

Sometimes the code inside the job returns an output. You can get the output of this code usingReceive-WorkCommand. EastReceive-WorkThe command takes a PSJob as input and writes the output of the job to the console. Everything that was generated while the job was running has been saved, so when the job is retrieved, everything that was saved at that time will be generated.

An example of this would be running the following code. This creates and starts a job that writesHello Worldfor the exit. It then retrieves the result of the job and posts it to the console.

$Job = Start-Job -ScriptBlock {Write-Exit 'Hallo Welt'}Receive-Job $Job

create scheduled jobs

Another way to interact with PSJobs is through aplanned work🇧🇷 Scheduled jobs are similar to a Windows scheduled task that can be configured withtask planner🇧🇷 Scheduled jobs create a way to easily schedule complex blocks of PowerShell scripts in a scheduled task. A scheduled job allows you to run a trigger-based PSJob in the background.

job trigger

Job triggers can be things like a specific time when a user logs in, when the system boots, and more. You can also repeat triggers at intervals. All these triggers are also definedNew-JobTriggerDomain. This command is used to specify a trigger that performs the scheduled work. A scheduled job without triggers must be run manually, but each job can have many triggers.

In addition to a trigger, you would still have a script block, just like a normal PSJob. If you have the activation and script blocks, I would use thisLog Scheduled JobCommand to create the job as shown in the next section. This command is used to specify attributes of the scheduled job, such as the script block to be executed and the triggers created with it.New-JobTriggerDomain.

Demonstration

Maybe you need a PowerShell code that runs every time someone logs on to a computer. You can create a scheduled job for this.

To do this, first define a trigger withNew-JobTriggerand set the scheduled job as below. This scheduled job writes a line to a log file every time someone logs on.

$Trigger = New-JobTrigger -AtLogon$Script = {"Benutzer $env:USERNAME angeleldet um $(Get-Date -Format 'y-M-d H:mm:ss')" | Out-File -FilePath C:\Temp\Login.log -Append}Register-ScheduledJob -Name Log_Login -ScriptBlock $Script -Trigger $Trigger

After executing the above commands, you will get output similar to creating a new job, showing the job ID, script block, and some other attributes, as shown below.

After a few login attempts, you can see on the screenshot below that the attempts have been logged.

use ofAlsJobParameter

Another way to use jobs is to useAlsJobParameter embedded in many PowerShell commands. Since there are many different commands, you can find them all withget commandAs shown below.

PS51> Get-Command -ParameterName AsJob

One of the most common commands iscall command🇧🇷 Normally, when you run this command, it immediately starts executing a command. While some commands return immediately so you can continue what you were doing, some wait for the command to finish.

Use ofAlsJobThe parameter does exactly what it sounds like and runs the executed command as a job instead of running it synchronously on the console.

While most of the timeAlsJobcan be used with local calculator,call commandit does not have a native option to run on a local machine. Is there a solution usinglocal serverwhile thecomputer nameparameter value. An example of this solution is shown below.

PS51> Invocar-Comando -ScriptBlock {Start-Sleep 5} -ComputerName localhost

to show youAlsJobParameters in action using the following examplecall commandsleep for five seconds and repeat the same command withAlsJobto show the difference in execution times.

PS51> Measure-Command {Invoke-Command -ScriptBlock {Start-Suspend 5}}PS51> Measure-Command {Invoke-Command -ScriptBlock {Start-Suspend 5} -AsJob -ComputerName localhost}

Runspaces: A bit like Jobs, but faster!

So far, you've only learned how to use additional threads with PowerShell using the built-in commands. Another option for multi-threading your script is to use a separate runspace.

performance spacesthey are the enclosed area in which the threads running PowerShell run. Although the runspace used with the PowerShell console is limited to a single thread, you can use additional runspaces to allow the use of additional threads.

Runspace x PSJobs

While a Runspace and a PSJob share many similarities, there are some key performance differences. The biggest difference between Runspaces and PS Jobs is the time it takes to set up and tear down each.

In the example in the previous section, the invocation of the created PS job took around 150ms. This is the best case scenario since the script block for the job did not contain much code and no additional variables were passed to the job.

Unlike PSJob creation, a runspace is created beforehand. Most of the time, the time required to start a runspace job is processed before any code is added.

Below is an example of running the same command we used for PSjob in runspace.

Instead, below is the code used for the runspace version. You can see that there is a lot more code to accomplish the same task. But the benefit of the additional code saves almost 3/4 of the time, so the instruction can be executed in 36 ms instead of 148 ms.

$Runspace = [runspacefactory]::CreateRunspace()$PowerShell = [powershell]::Create()$PowerShell.Runspace = $Runspace$Runspace.Open()$PowerShell.AddScript({Start-Sleep 5})$PowerShell. BeginInvoke()

Running Runtime Spaces: A Walkthrough

Using runspaces can be a daunting task at first, since you no longer need to handle PowerShell commands. You must deal directly with the .NET classes. In this section, we'll break down what it takes to create a runspace in PowerShell.

In this tutorial, you'll create a separate runspace for your PowerShell console and a separate instance of PowerShell. Then, assign the new runspace to the new instance of PowerShell, and add code to that instance.

Create the runspace

The first thing you need to do is create your new runspace. you do this to himrunspacefactoryClass. Store this in a variable as shown below so it can be referenced later.

$Runspace = [runspacefactory]::CreateRunspace()

Once the runspace is created, assign it to a PowerShell instance to run the PowerShell code. To do this, use thePower Shellclass and similar to runspace, you should store this in a variable as shown below.

$PowerShell = [Powershell]::Erstellen()

Then add the runspace to the PowerShell instance, open the runspace to run the code, and add the script block. This is shown below with a script block to sleep for five seconds.

$PowerShell.Runspace = $Runspace $Runspace.Open() $PowerShell.AddScript({Start-Sleep 5})

Run runspace

So far, the script block has not been executed. So far, everything has been defined for the runspace only. To run the script block, you have two options.

• get together()- It isget together()The method executes the script block in the runspace, but waits until the runspace returns before returning to the console. It's good to test this to make sure your code works correctly before letting it go.
• BeginInvoke()- Use ofBeginInvoke()The method is what you really want to see as a performance gain. This will start the execution of the script block in the runspace and take you back to the console immediately.

when you useBeginInvoke(), store the result in a variable, as it is needed to display the state of the script block in runspace, as shown below.

$Trabalho = $PowerShell.BeginInvoke()

After having the outputBeginInvoke()is stored in a variable, you can examine this variable to see the status of the job as shown belowcerradoProperty.

Another reason why you need to store the output in a variable is that unlikeget together()Method,BeginInvoke()it does not automatically return output when code completes. To do this, you must use theEndInvoke()method once completed.

In this example, there would be no output, but to end the call, you would use the following command.

$PowerShell.EndInvoke($Trabajo)

After all tasks queued in the runspace have completed, you should always close the runspace. This allows PowerShell's automated garbage collection process to clean up unused resources. Below is the command you would use to do this.

$ExecutionSpace.Close()

Using runspace groups

While using a runspace improves performance, it runs into a major single-threaded limitation. This is where runspace groups become important through the use of multiple threads.

In the previous section, you worked with only two runspaces. They only used one for the PowerShell console itself and the one they created manually. Runspace groups allow you to manage multiple runspaces behind the scenes using a single variable.

Although this multiple runspace behavior can be achieved with multiple runspace objects, using a runspace group makes administration much easier.

Runspace groups differ from individual runspaces in the way that they are configured. One of the main differences is that you define the maximum number of threads that can be used for the runspace pool. With a single runspace, you are limited to a single thread, but with a pool, you specify the maximum number of threads for the pool size.

The recommended number of threads in a runspace pool depends on the number of tasks to be performed and the computer on which the code is running. While increasing the maximum number of threads won't negatively affect speed in most cases, you might not see any benefit either.

Runspace Pool Speed ​​Demonstration

To show an example of where a group of runspaces outperforms a single runspace, you might want to create ten new files. If you were using a single runspace for this task, you would create the first file, then switch to the second, then the third, and so on until all ten have been created. The script block for this example might look like this. I would feed this block of scripts ten filenames through a loop and they would all be created.

$Scriptblock = { param($Name) New-Item -Name $Name -ItemType File}

The following example defines a script block that contains a short script that is given a name and creates a file with that name. A runspace pool is created with a maximum of 5 threads.

It then iterates through a loop ten times, each time assigning the iteration numberps🇧🇷 So you would have 1 in the first iteration, 2 in the second, and so on.

The loop creates a PowerShell object, assigns the block and script argument to the script, and starts processing.

Finally, at the end of the loop, it waits for all the queued tasks to complete.

$Scriptblock = { param($Name) New-Item -Name $Name -ItemType File}$MaxThreads = 5$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads)$RunspacePool.Open()$Jobs = @( )1..10 | Foreach-Object {$PowerShell = [powershell]::Create()$PowerShell.RunspacePool = $RunspacePool$PowerShell.AddScript($ScriptBlock).AddArgument($_)$Jobs += $PowerShell.BeginInvoke()}while ($ Jobs.IsCompleted -contém $false) {Start-Sleep 1}

Instead of creating threads one at a time, five threads are now created at a time. Without runspace groups, you would have to create and maintain five separate runspaces and five separatePower Shellinstances. This administration quickly spirals into chaos.

Instead, you can create a runspace pool, a PowerShell instance, and use the same code block and loop. The difference is that the runspace is sized to use only the five threads.

Create runspace groups

Creating a runspace group is very similar to the runspace created in a previous section. Below is an example of how to do this. Add a script block and the calling process is identical to a runspace. As you can see below, the runspace pool is created with a maximum of five threads.

$MaxThreads = 5$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads)$PowerShell = [powershell]::Create()$PowerShell.RunspacePool = $RunspacePool$RunspacePool.Open()

Comparison of runspaces and runspace groups for speed

To demonstrate the difference between a runspace and a runspace group, create and run a runspacestart sleepingorder from before. However, this time it needs to run 10 times. As you can see in the code below, it creates a runspace that sleeps for 5 seconds.

$Runspace = [runspacefactory]::CreateRunspace()$PowerShell = [powershell]::Create()$PowerShell.Runspace = $Runspace$Runspace.Open()$PowerShell.AddScript({Start-Sleep 5})1.. 10 | Foreach-Object { $Trabajo = $PowerShell.BeginInvoke() while ($Trabajo.IsCompleted -eq $false) {Start-Sleep -Millisekunden 100}}

Note that since you are using a single runspace, you must wait for it to complete before starting another call. Because of this, a 100ms idle state is added until the job is complete. While this can be reduced, you will see diminishing returns as you will spend more time checking if the job is complete than waiting for it to complete.

In the example below, you can see that it took around 51 seconds to complete 10 5-second sleep sets.

Instead of using a single runspace, now switch to a runspace group. Below is the code that is executed. You can see that there are some differences between using the two in the code below when using a runspace pool.

$RunspacePool = [runspacefactory]::CreateRunspacePool(1, 5)$RunspacePool.Open()$Jobs = @()1..10 | Foreach-Object { $PowerShell = [powershell]::Create() $PowerShell.RunspacePool = $RunspacePool $PowerShell.AddScript({Start-Sleep 5}) $Jobs += $PowerShell.BeginInvoke()}while ($Jobs. IsCompleted -contém $false) {Start-Sleep -Millisekunden 100}

As you can see below, this completes in just over 10 seconds, which is a vast improvement over the 51 seconds for the single runtime slot.

Below is a detailed summary of the differences between a runspace and a runspace group in these examples.

Easier in runspaces with PoshRSJob

When programming, it often happens that you do something more comfortable and accept the small loss of performance. This could be because it makes the code easier to write or read, or it could simply be a matter of preference.

The same goes for PowerShell, where some people use PSJobs instead of Runspaces for ease of use. There are a few things you can do to split the difference and get the best performance without making it too cumbersome to use.

There is a widely used module calledPoshRSTrabajowhich contains modules in the style of regular PSJobs, but with the added benefit of using runspaces. Instead of providing all the code to create the runspace and powershell object, the PoshRSJob module does all of that when you run the commands.

To install the module, run the following command in a PowerShell administrative session.

PoshRSJob installation module

Once the module is installed, you can see that the commands are the same as the PSJob commands with aRSPrefix. backwards ofStart-JobThat's itHome-RSJob🇧🇷 Upside downget a jobThat's itGet RSJob.

Below is an example of the same command running on a PSJob and again on an RSJob. As you can see, they have very similar syntax and output, but they are not exactly the same.

Below is some code that can be used to compare the speed difference between a PSJob and an RSJob.

Medida-Comando {Start-Job -ScriptBlock {Start-Sleep 5}}Medida-Comando {Start-RSJob -ScriptBlock {Start-Sleep 5}}

As you can see below, there is a big difference in speed since RSJobs still uses hidden runspaces.

Foreach Object -Parallel

The PowerShell community wanted a simpler, more integrated way to quickly multi-thread a process. The result is the parallel key.

As of this writing, PowerShell 7 is still in preview, but they added aparallelparameters forfor each objectDomain. This process uses execution spaces to parallelize the code and uses the script block provided for thefor each objectas a script block for the runspace.

While the details are still being worked out, this could be an easier way to use runspaces in the future. As you can see below, you can go through various stages of sleep very quickly.

Messbefehl {1..10 | Foreach-Object {Start-Sleep 5}}Measure-Befehl {1..10 | Foreach-Object -Parallel {Start-Sleep 5}}

Multi-threaded challenges

While multithreading has sounded awesome so far, it's not exactly the case. There are many challenges that come with multi-threaded code.

Use of variables

One of the biggest and most obvious challenges with multi-threading is that you can't share variables without passing them as arguments. There is the only exception to asynchronized hash tableBut that's a conversation for another day.

PSJobs and Runspaces work without access to existing variables and there is no way to interact with variables used in different runspaces on your console.

This presents a major challenge for dynamically passing information to these jobs. The answer varies depending on the type of multithreading you are using.

For himStart-JobmiHome-RSJobfrom the PoshRSJob module you can use theargument list-Parameters to provide a list of objects that are passed as parameters to the script block in the order that it lists them. Below are examples of the commands used for PSJobs and RSJobs.

PWorked:

Start-Job -Scriptblock {param ($Text) Write-Output $Text} -ArgumentList "¡Hola Welt!"

RNeed:

Start-RSJob -Scriptblock {param ($Text) Write-Output $Text} -ArgumentList "¡Hola Welt!"

Native runspaces don't offer the same ease. Instead, you should use theadd argument()method on the PowerShell object. Below is an example of what it would look like for each.

Execution Space:

$Runspace = [runspacefactory]::CreateRunspace()$PowerShell = [powershell]::Create()$PowerShell.Runspace = $Runspace$Runspace.Open()$PowerShell.AddScript({param ($Text) Write-Output $ Texto})$PowerShell.AddArgument("¡Hola, Welt!")$PowerShell.BeginInvoke()

Although runspace groups work the same way, here is an example of how to add an argument to a runspace group.

$MaxThreads = 5$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads)$PowerShell = [powershell]::Create()$PowerShell.RunspacePool = $RunspacePool$RunspacePool.Open()$PowerShell.AddScript({param ($Texto) Schreibausgabe $Texto})$PowerShell.AddArgument("¡Hola, Welt!")$PowerShell.BeginInvoke()

Login

Multithreading also presents registration challenges. Since each thread works independently, they cannot log on to the same location. If you tried to write a file with multiple threads, when one thread wrote to the file, no other thread could. This can slow down your code or cause it to fail altogether.

As an example, the following is code that tries to check in 100 times on a single file with 5 threads in a runspace pool.

$RunspacePool = [runspacefactory]::CreateRunspacePool(1, 5)$RunspacePool.Open()1..100 | Foreach-Object {$PowerShell = [powershell]::Create().AddScript({'Hello' | Out-File -Append -FilePath .\Test.txt})$PowerShell.RunspacePool = $RunspacePool$PowerShell.BeginInvoke() }$RunspacePool.Cerrar()

In the output, you don't see any errors, but if you look at the size of the text file, you see below that not all 100 jobs completed successfully.

Get-Content-Path .\Test.txt).Count“ class=“wp-image-3241″/>

One way around this is to log into separate files. This gets rid of the file-locking problem, but you have a lot of log files that you'd have to go through to figure out everything that happened.

Another alternative is to allow some output's timing to be disabled and just log what a job did after it completed. That way, you can have everything serialized in your original session, but you lose some details since you don't necessarily know what order everything happened in.

abstract

While multithreading can bring huge performance gains, it can also be a pain in the ass. While some workloads benefit greatly, others may not benefit at all. There are many advantages and disadvantages to using multithreading, but when used correctly, it can drastically reduce the execution time of your code.

Other reading

• PowerShell 7 Preview 3: When was born -Parallel
• How to create a PowerShell GUI for your scripts