how to Invoke-Webrequest to download CSV file which is random tempname.csv

Welcome Forums General PowerShell Q&A how to Invoke-Webrequest to download CSV file which is random tempname.csv

This topic contains 9 replies, has 4 voices, and was last updated by

 
Participant
3 months, 4 weeks ago.

  • Author
    Posts
  • #130479

    Participant
    Points: 90
    Rank: Member

    Hi,
    We have a website , its a reporting site where I can filter/bookmark the filters on conditions and it creates an tabular output on the page.
    I can right click the page and select export to export the data to csv. Every time I do that it generates a random name for the csv file like ae7bxtjs84snbdwwemf845d.csv  and I do a save as to change it to desired filename.

    the question is how do I use Invoke-Webrequest $URL -Outfile C:\xyz.csv

    if i use the same temp name, it works for a while and later downloads the empty file. Probably the cookie session expired or temp name expired.

    I cannot share the URL since it is official. Is there any example that you can help me or suggest to work towards?

  • #130485

    Participant
    Points: 639
    Helping Hand
    Rank: Major Contributor

    It's probably not going to be that straight-forward. You can logon to the page with Invoke-WebRequest and persist the session with the WebSession parameter. Open up a browser, like Chrome, hit F12 for Developer Tools and go to Network. Then watch the URL's and the body information that is being passed to the browser, which you emulate with Invoke-WebRequest.

    With that said, when you do that, the CSV is being generated Server-Side, not in the browser. You have two basic options:

    • Generate the HTML table and then parse that table.
    • Emulate the entire process and click the filters and the buttons to see if you can generate the CSV.

    It's hard to help unless you see the calls that are being made.

    • #130613

      Participant
      Points: 90
      Rank: Member

      Thanks for the suggestion. yes, I did follow the F12 path...
      The URL doesnt change untill the csv export button is clicked and that is what confuses me.
      The webpage is qlikview webpage hosted on a server.
      Lets say the webpage is qlikview.com/someheaderparameters.
      I select filters like manager name, server category... so far the URL remains the and only the table content changes inside the page.
      I right click anywhere on the content and select export ..
      Now the URL changes to qlikview.com/someheaderparameters/tempname.csv and I get save / saveas option in the browser bottom.

      When I look into the body of this URL , it contains the comma seperated values that is displayed in the page.
      The body of the URL before that seems to contain a $URL session value and a Temp name generated by calling a javascript internally..

      Even when I change the filters, the URL doesn't change until I select export... And then I see the same session value with a new Tempname.csv.

      Each filters creates a new tempname and if I go back to previous filters it creates the same tempname again untill the session expires.
      Looks complicated to me 🙂 🙂

       

    • #130617

      Participant
      Points: 639
      Helping Hand
      Rank: Major Contributor

      If the url stays the same, then most likely a POST is being performed with a Body. Assuming you are using Chrome, hit F12 and got to Network > XHR. When you do something the page, you should see a POST and you can look at the Request Headers and Request Payload (a.k.a Body). To test, you can just grab the same JSON and try posting it, something like this:

      #What you saw in Request Payload
      $jsonBody = @"
      {
          "filter":  {
                         "Manager":  "John",
                         "Category":  "Awesome"
                     }
      }
      "@
      
      $params = @{
          Uri = 'www.qlikview.com/someheaderparameters' 
          Body = $jsonBody 
          Method = 'POST'
          ContentType = 'application/json'
      }
      
      Invoke-WebRequest @params
      

      When you know it's working, you can generate a PSObject that would allow you to build a dynamic query with something like this:

      $filter = [pscustomobject] @{
          filter = [pscustomobject] @{
              Manager = 'John'
              Category = 'Awesome'
          }
      }
      
      $jsonBody = $filter | convertTo-Json
      
      $params = @{
          Uri = 'www.qlikview.com/someheaderparameters' 
          Body = $jsonBody 
          Method = 'POST'
          ContentType = 'application/json'
      }
      
      
      Invoke-WebRequest @params
      

      Another place to look is if the vendor has documentation on the API:

      https://help.qlik.com/en-US/qlikview-developer/November2017/Content/APIsAndSDKs.htm

      Then you can even search to see if there are examples of Powershell:

      https://community.qlik.com/t5/QlikView-Documents/QlikView-and-Windows-PowerShell/ta-p/1497315

    • #130761

      Participant
      Points: 90
      Rank: Member

      It wasnt json response after all. The links you gave for reference looks good but the API docs are in qvw extension. I donot have qlikview software installed . just using the web page.. I donot think I can open it.. Is there any other way?

      By the way, i took notes of the XHR details and took a pic for reference.. . Check if it helps anyway to guide me..

      If cannot, then I started getting thoughts of skipping it 🙁

      How do I attach images here? ..

       

       

  • #130509

    Participant
    Points: 1,317
    Helping Hand
    Rank: Community Hero

    What have you tried thus far?
    Show you code attempt?

    You say...

    'I can right click the page and select export to export the data to csv.'

    SO, you mean you can Right Click anywhere on the page and you get the export option?
    So, this says to me that there is code on that page that allows this.

    Therefore, this...

    Invoke-Webrequest $URL -Outfile C:\xyz.csv

    … will not do this. You have to use web page navigation and UI automation in this use case. You scrape the page to get all the elements and then work with those in code.

    $w = Invoke-WebRequest -Uri 'https://powershell.org'
    $w | Get-Member
    $w.StatusCode
    $w.AllElements
    $w.AllElements.Count
    $w.Links.Count
    $w.Links
    $w.Forms
    $w.Forms[0].Fields
    $w.RawContent
    $w.ParsedHtml
    
    • #130616

      Participant
      Points: 90
      Rank: Member

      As I mentioned in my reply above. I tried to go through the F12 options to identify where it is navigating.. but that didnt help much.

      QlikViewPageSample

      This is the sample page where left sides are filters and the data at the bottom part of the page. I can right click anywhere in the data and choose export excel or csv.

      I tried all that you mentioned above.. the problem is since the main URL is not changing, I do not know where the session value and temp value is getting generated and I see its calling a javascript in the previous step.

      However, since the URL is not changing, the response with your codes above remains same as in the beginning.
      Talking about UI automation, Yes, the same is achievable to RPA(Robotic Process Automation) and UIPath.

      However, I want to do it in powershell 🙂

  • #130824

    Participant
    Points: 380
    Helping Hand
    Rank: Contributor

    Why not check with Qlik?

    If they can direct you to the correct API-call etc. for what you want to do?
    It will be alot more predictive than trying to use web page navigation.

    • #130908

      Participant
      Points: 90
      Rank: Member

      Good point.. Working in corporate, you end up working with support Line Of Business who works with vendor. Unfortunately, I cannot ask them to reach out to vendor to ask for API's. Or I am sure the internal team has the documentations. However, to open them I cannot get Qlik client installed on my desktop due to policy and scope restrictions..

      Anyways, I do not want to dig deep into why what when.. .

      Thanks for all your responses. I will find a way through IE DOM Explorer and use COM object to achieve it.

    • #130953

      Participant
      Points: 380
      Helping Hand
      Rank: Contributor

      If it's not a closed API then the whole point of it is to be able to consume the data without a specific client.
      E.g. using powershell, python, java or whatever as long as it can generate the call that the API requires.

      Also Qlik have their own forums so maybe use that if you don't want to open a support case.
      https://community.qlik.com/t5/Qlik-Sense-Integration/bd-p/qlik-sense-integration-extension-api

      It's of course up to you how you want to proceed but web navigation will most likely not be that reliable.
      Good luck.

The topic ‘how to Invoke-Webrequest to download CSV file which is random tempname.csv’ is closed to new replies.

denizli escort samsun escort muğla escort ataşehir escort kuşadası escort