How do I split large folder into smaller folders?

This topic contains 14 replies, has 2 voices, and was last updated by  Cohen Carryl 5 days, 17 hours ago.

  • Author
    Posts
  • #87554

    Cohen Carryl
    Participant

    Can someone please help me with a PowerShell script.

    I want to split a larger folder into a smaller folder.

    With 75 files within a folder. The folder size limit should not exceed 800MB.

    Thanks in advance.

  • #87557

    Cohen Carryl
    Participant

    $filesperfolder = 75
    $sourcePath = "C:\Source\SPLITFILELOCATION\"
    $destPath = "C:\Source\Move Here"
    $i = 0;
    $folderNum = 1;

    Get-ChildItem "$sourcePath\*.pdf " | % {

    New-Item -Path ($destPath + "\" + $folderNum) -Type Directory -Force
    Move-Item $_ ($destPath + "\" + $folderNum);

    $i++;

    if ($i -eq $filesperfolder){
    $folderNum++;
    $i = 0 ;
    }
    }

    • #87580

      Jeremy Corbello
      Participant

      From what I can see right off the bat, you're recreating the folder with every iteration of the foreach loop. I'd put a test in there to see if the folder exists first. The rest of it looked pretty good.

      $filesperfolder = 75
      $sourcePath = "C:\Source\SPLITFILELOCATION\"
      $destPath = "C:\Source\Move Here"
      $i = 0
      $folderNum = 1
      
      Get-ChildItem "$sourcePath\*.pdf " | ForEach-Object {
          if (!(Test-Path "$destPath\$folderNum")) {
              New-Item -Path "$destPath\$folderNum" -Type Directory -Force
              }
          Move-Item $_ "$destPath\$folderNum"
          $i++
          if ($i -eq $filesperfolder){
              $folderNum++
              $i = 0 
          }
      }
  • #87586

    Cohen Carryl
    Participant

    Hi Jerry,

    Can you modify the script to include your suggestion?

    Thank you,

    Cohen

    • #87593

      Jeremy Corbello
      Participant

      I did, that's the code that is in my comment above.

    • #87595

      Cohen Carryl
      Participant

      My apologies. I read your comment, but I didn't look at the script.

      Is there a way to keep the files in the source path location and copy it to the destination without erasing the files from source path?

    • #87598

      Jeremy Corbello
      Participant

      Absolutely! Use Copy-Item instead of Move-Item. Everything else stays the same:

      $filesperfolder = 75
      $sourcePath = "C:\Source\SPLITFILELOCATION"
      $destPath = "C:\Source\Move Here"
      $i = 0
      $folderNum = 1
      
      Get-ChildItem "$sourcePath\*.pdf " | ForEach-Object {
          if (!(Test-Path "$destPath\$folderNum")) {
              New-Item -Path "$destPath\$folderNum" -Type Directory -Force
              }
          Copy-Item $_ "$destPath\$folderNum"
          $i++
          if ($i -eq $filesperfolder){
              $folderNum++
              $i = 0 
          }
      }
  • #87601

    Cohen Carryl
    Participant

    Thank you, Jerry!

    Your help is very much appreciated.

    Cheers,

    Cohen

  • #87610

    Cohen Carryl
    Participant

    So could there be a combination of a folder name and folder number?

    folder name jerry folder number 1 = jerry 1

    Our user a folder name Jerry 1
    folder name Jerry 2 and so on.

    A folder name would be ideal rather than the folder number, my manager may get confused with the name convention.

    • #87613

      Jeremy Corbello
      Participant

      Sure. Look below where New-Item and Copy-Item are.

      When you are making a string on the fly, you can mix characters and variables.

      $filesperfolder = 75
      $sourcePath = "C:\Source\SPLITFILELOCATION"
      $destPath = "C:\Source\Move Here"
      $i = 0
      $folderNum = 1
      
      Get-ChildItem "$sourcePath\*.pdf " | ForEach-Object {
          if (!(Test-Path "$destPath\Folder$folderNum")) {
              New-Item -Path "$destPath\Folder$folderNum" -Type Directory -Force
              }
          Copy-Item $_ "$destPath\Folder$folderNum"
          $i++
          if ($i -eq $filesperfolder){
              $folderNum++
              $i = 0 
          }
      }
    • #87614

      Jeremy Corbello
      Participant

      And to be fair, I didn't write it like I normally do. Here's how I would typically create a string out of a mix of variables and characters. Not sure if it's best practice or not, just how I typically do it.

      $filesperfolder = 75
      $sourcePath = "C:\Source\SPLITFILELOCATION"
      $destPath = "C:\Source\Move Here"
      $i = 0
      $folderNum = 1
      
      Get-ChildItem "$($sourcePath)\*.pdf " | ForEach-Object {
          if (!(Test-Path "$($destPath)\Folder$($folderNum)")) {
              New-Item -Path "$($destPath)\Folder$($folderNum)" -Type Directory -Force
              }
          Copy-Item $_ "$($destPath)\Folder$($folderNum)"
          $i++
          if ($i -eq $filesperfolder){
              $folderNum++
              $i = 0 
          }
      }
  • #87874

    Cohen Carryl
    Participant

    Thank you, Jeremy!

  • #87877

    Cohen Carryl
    Participant

    Jeremy, can you assist me fixing this script?

    The client started sending the files zipped multiple time so my script is not unzipping it the way it wants to.

    The goal is to unzip all the files in that particular directory and send out an email with the content of the zip file.

    Example

    You received the files below

    happyone.err
    happytwo.out
    happythree.err

    Another issue I noticed, I was trying to put something in place that runs from 4 pm on Tuesday. If no files were received for the day, sent out a notification making it be known.

    The client actually sent files anytime.

    Your help will be gladly appreciated again.

    The email and password and username in the script were changed for security purpose, I wouldn't want to get in trouble for giving out company credentials.

    This is the script:

    #powershell Set-ExecutionPolicy RemoteSigned

    $Username = "Outbound@companyemail.com";
    $Password= "Xova0000000";
    $nl = [Environment]::NewLine

    function Send-ToEmailSource([string]$email, $foldername){

    $message = new-object Net.Mail.MailMessage;
    $message.From = "Outbound@companyemail.com";
    $message.To.Add($email);
    $FileNames = Get-ChildItem -Path $foldername -Recurse
    $FileNames = $FileNames | ForEach-Object {"$_$nl"}
    $message.Subject = "$foldername folder received";
    if($FileNames.length -gt 0){
    $message.Body = "$FileNames files are present in the source folder";
    }else{
    $message.Body = "There are no err,out files are present in the source folder";
    }
    $smtp = new-object Net.Mail.SmtpClient("smtp.office365.com", "587");
    $smtp.EnableSSL = $true;
    $smtp.Credentials = New-Object System.Net.NetworkCredential($Username, $Password);

    $smtp.send($message);

    }

    function Send-ToEmailTarget([string]$email, $foldername){

    $message = new-object Net.Mail.MailMessage;
    $message.From = "Outbound@centaurihs.com";
    $message.To.Add($email);
    $FileNames = Get-ChildItem -Path $foldername
    $FileNames = $FileNames | ForEach-Object {"$_$nl"}
    $message.Subject = " folder was copied to BCBS AZ"
    if($FileNames.length -gt 0){
    $message.Body = "$FileNames The files above were copied in the BCBS AZ target folder";
    }else{
    $message.Body = "There are no err, out files copied in the target folder";
    }
    $smtp = new-object Net.Mail.SmtpClient("smtp.office365.com", "587");
    $smtp.EnableSSL = $true;
    $smtp.Credentials = New-Object System.Net.NetworkCredential($Username, $Password);
    $smtp.send($message);

    }

    $srcPath = "\\DEVOPS01\Users\devops.service"
    $targetPath = "\\DEVOPS01\Users\devops.service\ABC"
    $fname = (get-date).ToString("yyyMMdd")
    $sourceFolder = "$srcPath\$fname"
    #$targetFolder = "$targetPath\$fname"

    Add-Type -AssemblyName System.IO.Compression.FileSystem
    function Unzip
    {
    param([string]$zipfile, [string]$outpath)
    [System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
    }

    if (Test-Path $sourceFolder ){
    $Files = Get-ChildItem $sourceFolder\*.* -include *.zip,*.out,*.err -Recurse
    $srcZipFile = $Files | where {($_.extension -eq ".zip") }#Identifying zip file in the source path#

    if ((Test-Path $sourceFolder\*.out) -or (Test-Path $sourceFolder\*.err) -or (Test-Path $sourceFolder\*.zip)){

    Send-ToEmailSource -email "cohen.carryl@companyemail.com" $sourceFolder

    if (Test-Path $srcZipFile){
    $temp = $([io.fileinfo]"$srcZipFile").basename
    $unzipedFolderPath = "$sourceFolder\$temp"
    Unzip "$srcZipFile" "$unzipedFolderPath"
    $tempList = get-childitem $unzipedFolderPath -recurse
    $zipFileList = $tempList | where {($_.extension -eq ".err") -or ($_.extension -eq ".out") }

    if ($zipFileList.length -gt 0 ){
    Copy-Item $unzipedFolderPath\*.err $targetPath\ -Force
    Copy-Item $unzipedFolderPath\*.out $targetPath\ -Force
    Remove-Item $unzipedFolderPath -Force -Recurse

    }else{
    Remove-Item $unzipedFolderPath -Force -Recurse

    }
    }
    Copy-Item $sourceFolder\*.err $targetPath\ -Force
    Copy-Item $sourceFolder\*.out $targetPath\ -Force

    Send-ToEmailTarget -email "cohen.carryl@companyemail.com" $targetPath

    }
    }else{

    Write-Host "There is no files received today yet"
    }

    • #87902

      Jeremy Corbello
      Participant

      Can you clarify this:

      The client started sending the files zipped multiple time so my script is not unzipping it the way it wants to.

      Are you saying that they are zipping a file, then zipping that zipped file? i.e. nested zips?

      And what was the issue here? Sorry, just not really understanding what you're needing:

      Another issue I noticed, I was trying to put something in place that runs from 4 pm on Tuesday. If no files were received for the day, sent out a notification making it be known.

  • #87908

    Cohen Carryl
    Participant

    This is an example.

    Client would files to this path. \\Example\Zipfile\Client name\

    My Script looks for zip file if it exists if not extract the file.

    It's look for yyymmdd

    This how they use to send it.

    \\Example\Zipfile\Client name\20171208

    The script would see the files in the folder and transfer it to the target destination.

    This is how they are sending it now:

    \\Example\Zipfile\Client name\20171208\20171208.zip\20171207\with files inside

    \\Example\Zipfile\Client name\20171208\20171208.zip\20171207-processed\with files inside.

    I want to -Recurse or -Force or both and look for file extension .err and .out and send out 1 email notification with the .err and .out in the folder path and the subdirectories \\Example\Zipfile\Client name\20171208\

You must be logged in to reply to this topic.