Best way to build a collection out of MSMQ

This topic contains 6 replies, has 2 voices, and was last updated by Profile photo of Troy Helms Troy Helms 3 years, 2 months ago.

  • Author
    Posts
  • #14179
    Profile photo of Troy Helms
    Troy Helms
    Participant

    Here is my challenge. I've taken a script that used to take a couple hours to process. Now, by leveraging jobs, I was able to reduce it to 2 minutes. Since this script does a large amount of output, (I previously wrote all of the output to flag csv files, then read them all in and the end and generated reports from them), I leveraged MSMQ to dump all of the out from the jobs. My intent is to come back at the end after all the jobs complete, read in each of the queues and create my report.

    I have read that using += to add a large number of objects to an array is a bad idea that gets exponentially slower with a large amounts of data.

    My questions is, what would be the best way to read in all of the objects from the queue so I can build a report? Before I read in a csv, used ConvertTo-HTML, and away I went.
    I know I would have to build a loop to pull out each message to add it to a table or some such.

    Any thoughts?

  • #14180
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    There are a couple of options. You can use the .NET List or ArrayList classes, or you can create a PowerShell function that just ouputs the objects, and let the engine take care of turning it into a collection. Here are examples of each:

    $arrayList = New-Object System.Collections.ArrayList
    for ($i = 0; $i -lt 10000; $i++)
    {
        $null = $arrayList.Add($i)
    }
    
    $array1 = $arrayList.ToArray()
    
    function Test
    {
        for ($i = 0; $i -lt 10000; $i++)
        {
            $i
        }
    }
    
    $array2 = Test
    

    In general, the second option will give better performance, because most of the work is performed by compiled .NET code, but both are much better than using += on arrays.

  • #14182
    Profile photo of Troy Helms
    Troy Helms
    Participant

    Sounds like the second method is what I want to use, just not sure how to implement it with what I have.

    For example, here is out the object is created:

    [xml]$UpdatedUsers = “
    $Username
    $FirstName
    $LastName
    $Title
    $yEmail
       

    Now, to get the few thousand objects back from MSMQ, I can do something like:

    $UserInfo= (Receive-MSMQueue -Name "UpdatedUsers" -Private -TargetType XML)
    

    That gives me 1 xml object back from the queue into an object. I'm stuck on how to repeat that till the queue is empty and have one object I can plug into my report I'm creating. Before I just used += (which gets ugly when it gets big).

    Any help is appreciated.

  • #14185
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    What happens when the queue is empty and you call Receive-MSMQueue? Does it just return $null, or does it produce an error?

    In this case, you don't even need to have a function, if you don't want to. The results of a loop can be assigned directly to a variable, like this (assuming, for this example, that a return value of $null is what happens when there's nothing left in the queue):

    $UserInfo = while ($true)
    {
        $object = Receive-MSMQueue -Name "UpdatedUsers" -Private -TargetType XML
        if ($null -eq $object)
        {
            break
        }
        else
        {
            $object
        }
    }
    
  • #14186
    Profile photo of Troy Helms
    Troy Helms
    Participant

    Yes, it returns $Null when it's empty. I think this is exactly what I'm looking for. I'll plug this in on Monday and let you know how it goes. Thanks so much, you just made my weekend! 🙂

  • #14187
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    No problem! 🙂 Here's a shorter way of writing that, if you don't mind assigning variables inside your loop condition:

    $UserInfo = while ($null -ne ($object = Receive-MSMQueue -Name "UpdatedUsers" -Private -TargetType XML))
    {
        $object
    }
    

    The end result is the same.

  • #14202
    Profile photo of Troy Helms
    Troy Helms
    Participant

    I admit, I couldn't wait till Monday. I got excited about it and worked on it yesterday and have it up and running. Still need to handle a lot of special error handling/checks, but overall, the script looks like it will be a success! And it's SO much faster!
    Previously it took 30+ minutes to process 500 users with no data changes (and considerably longer when it actually had to update data). In my test runs, it's processing 1500 users, updating over 700 of them, and takes < 3 minutes. That's HUGE. I have encountered a couple anomalies with Microsoft's Active Directory Module though. Seems to give errors when too many jobs are calling Get-ADUser at the same time. I worked around it, but it's an annoying little bug for sure. I thought about seeing if I could do what I need in .Net vs using the module to not have to deal with it at all. Thanks again for your help! You have made my Monday!

You must be logged in to reply to this topic.