DNS records and redirect lookup combined – Help needed

Welcome Forums General PowerShell Q&A DNS records and redirect lookup combined – Help needed

Viewing 5 reply threads
  • Author
    Posts
    • #196970
      Participant
      Topics: 8
      Replies: 9
      Points: 102
      Rank: Participant

      Hi everyone!

      I am using powershell to look up large amounts of domains at the same time, and getting their Origin (local file), Nameserver, A-record and MX-records.

      This is working fine, but i am trying to figure out if i can add the ability for the lookup, to find out if the domains are redirected.

      Currently the output of the script gives me this:

      “Origin”,”DomainName”,”Resolved”,”NameServer”,”A-Record”,”MX-Record”

      “C:\Users\jonat\Desktop\DNS PROCCESSING\Batchtest\Scanning-test.txt”,”testdomain.dk”,”True”,”ns2.domaintest.dk, ns.domaintest.dk”,”87.238.244.22″,”alllogistics.skyhost.test”

      My goal is to modify it to this:

      “Origin”,”DomainName”,”Resolved”,”NameServer”,”A-Record”,”MX-Record”,“RedirectType”,”RedirectDomainName”

      “C:\Users\jonat\Desktop\DNS PROCCESSING\Batchtest\Scanning-test.txt”,”testdomain.dk”,”True”,”ns2.domaintest.dk, ns.domaintest.dk”,”87.238.244.22″,”alllogistics.skyhost.test,301,development.testdomain.dk”

      I am using powershell for some practical tasks, but im out of bounds here.
      The script i am using on the link below, and i hope if nothing else, that someone else can gain some value from it 🙂

      Found on Github Gist here:
      https://gist.github.com/Jonatantwn/378d7c7b8be503bca223b6438014eb57

      Id really appreciate some help! 🙂

      Hopeful regards
      Jonatan

    • #197288
      Senior Moderator
      Topics: 5
      Replies: 133
      Points: 706
      Helping Hand
      Rank: Major Contributor

      This should be doable, but it might not be simple.

      Adding the object properties for the extra information is relatively easy. You’re already creating a custom object, so just add new lines for the additional properties.

      Discovering the redirection is more complicated. There isn’t an existing cmdlet specifically for this, so you’ll have to build your own function to get the data you want.
      This example uses the WebRequest class: https://4sysops.com/archives/how-to-test-for-a-http-url-redirect-in-powershell/
      This example uses the Invoke-WebRequest cmdlet: https://winblog.it.umn.edu/2018/05/19/getting-redirected-uris-in-powershell/
      Technically these are the same thing as PowerShell cmdlets are just abstractions for .Net classes, but the implementation is different. Using System.Net.WebRequest may offer more flexibility but will require some understanding of .Net (outside of PowerShell). The second example seems very straightforward.

      Either way, you could use this methodology to test whether a given URL is redirected and what the redirect target is. However, neither example captures the type of redirect, and I haven’t found anything helpful for that so far.

      • This reply was modified 6 months, 3 weeks ago by grokkit.
    • #197741
      Participant
      Topics: 8
      Replies: 9
      Points: 102
      Rank: Participant

      Hi again!

      I managed to actually make it work, with Invoke-Webrequest, with a bunch of not so pretty filters and extensions.

      However, works.

      But, i can get the function i made called “Rmatch” to work.

      I want it to just get the value (that it prints correcntly to the CSV) and match is with the domain name.
      This figures out, if the domain points “towards itself” or towards something else with a simple True/false return.

      Can you see what im doing wrong here?

      New Gist link for Vol. 2 :

      https://gist.github.com/Jonatantwn/8813f274f56e549b27c18a932b940c4c

    • #197747
      Participant
      Topics: 13
      Replies: 1683
      Points: 2,849
      Helping Hand
      Rank: Community Hero

      It appears you are trying to get a boolean value (True\False) in RMatch. The $Domainfinal is not defined, so it’s NULL. You have two options:

      Option 1: Define $DomainFinal

      ...
      try {
          $NSRecords = Resolve-DnsName $DomainName -Type NS -EA 1 | Where-Object { $_.QueryType -eq 'NS' }
          $DomainFinal = (Invoke-WebRequest -Uri $DomainName -UseBasicParsing).BaseResponse.ResponseUri.AbsoluteUri -replace "https://" -replace "http://" -replace "www." -replace ".dk/", ".dk"  -replace ".com/", ".com"
          [PSCustomObject]@{
              Resolved    = $true
              Rmatch = $DomainName -in $Domainfinal
              DomainName  = $DomainName
              Domainfinal = $DomainFinal
              NameServer  = $NSRecords.NameHost
              'A-Record'  = (Resolve-DnsName $DomainName -Type A).IPAddress
              'MX-Record' = (Resolve-DnsName $DomainName -Type MX).NameExchange
              Origin      = $FileName
      
          }
      ...
      

      Option 2: Use the calculated expression

      $myOutput | Select-Object Origin, 
                                Resolved, 
                                DomainName,
                                @{n='Domainfinal'; e={$_.'Domainfinal' -join ', '}},
                                @{n='NameServer';e={$_.NameServer  -join ', '}},
                                @{n='Rmatch';  e={$_.'Rmatch'  -in @($_.DomainFinal)}},
                                @{n='A-Record';  e={$_.'A-Record'  -join ', '}},
                                @{n='MX-Record'; e={$_.'MX-Record' -join ', '}}
      

      Also, based on the -joins, you would or could have multiple domains to validate against, so you need to use -contains or -in to see if the domain is in the array. It’s difficult to assist fully as there is no data to test, but something like so:

      PS C:\Users\rasim> 'yahoo' -in 'google','yahoo','msn'
      True
      
      PS C:\Users\rasim> 'microsoft' -in 'google','yahoo','msn'
      False
      
      PS C:\Users\rasim> 'google','yahoo','msn' -contains 'yahoo'
      True
      PS C:\Users\rasim> 
      
    • #197756
      Senior Moderator
      Topics: 5
      Replies: 133
      Points: 706
      Helping Hand
      Rank: Major Contributor

      More specificity on how Rmatch is not working would be useful, but I’m assuming that it is empty every time. Your Rmatch object property depends on the values of DomainName and Domainfinal, but you are setting Rmatch before you set values for DomainName and Domainfinal, which would result in Rmatch being set using empty variables. If Rmatch is always empty, this is probably why.

      Also, you should probably read the documentation on matching operators. The -like operator is more appropriate here than -match because you’re not using regex, you just want a direct string comparison.

      *Edit* Also, as Rob suggested, a sample of your input file and CSV output would be useful for troubleshooting.

      • This reply was modified 6 months, 3 weeks ago by grokkit.
    • #197852
      Participant
      Topics: 8
      Replies: 9
      Points: 102
      Rank: Participant

      Hi Rob and Grokkit!

      Thanks alot for your input.
      I am as you have likely guessed a novice here, so your help is greatly appreciated!
      I am still working on understanding Powershell syntax.

      Regarding the input format, it is very simple. I just add a .txt file with a list of domains to the “Batchtest” folder, like so:
      Domain1.com
      Domain2.com
      Domain3.com

      Here you can see a screenprint of the output made, after i added Rob Simmers “Option 1 solution” to the code snippet.
      https://imgur.com/a/9G9nMb0 – I have had to redact some information, as i am using some company locations to test with.
      The domains to vary from the “DomainName” and “Domainfinal”  and the system now works to identify non-matching domains correctly!
      This was also possible before, but the True/False

      I thought that i would explain the purpose of the system briefly:
      The goal of the system is to scan through a large database of domains, where many are inactive or are redirects, thus only having need for a simple hosting platform, and not the processing power of a website.
      It is a cleanup and webhosting-economic operation, as a lot of inactive domains have not been cleaned out properly for quite some time.

      Thanks a lot for your help, and have a great day!

      Best regards
      Jonatan

Viewing 5 reply threads
  • The topic ‘DNS records and redirect lookup combined – Help needed’ is closed to new replies.