Updated readme and added raw file.

README.md

@@ -1,48 +1,40 @@
 # RobotsDisallowed
-The RobotsDisallowed project is a harvest of the Disallowed directories from the robots.txt files of the world's top websites--specifically the Alexa 100K.
 
-This list of Disallowed directories is a great way to supplement content discovery during a web security assessment, since the website owner is basically saying "Don't go here; there's sensitive stuff in there!".
+The RobotsDisallowed project is a harvest of the robots.txt disallowed directories of the world's top websites---specifically thos of the Alexa 100K and the Majestic 100K.
 
-It's basically a list of potential high-value targets.
+This list of Disallowed directories is a great way to supplement content discovery during a web security assessment or bug bounty, since the website owner is basically saying, 
 
-## The project
-
-So what we did is take the Alexa Top 100,000 websites, download their robots.txt files, extracted all Disallowed directories, and then performed a bunch of cleanup on them (they are a mess) to make the lists as useful as possible during web assessments.
-
-## How to use the project
-
-You use the project by coming to the root and downloading the DisallowedDirectories files there. You can then plug them into your favorite web assessment tool/function, e.g., Burp Intruder.
+> "Don't go here; there's sensitive stuff in there!
 
-The files are broken down into Top-n lists, which are sorted lists based on the most common directories found. But if you are pressed for time or are looking for the highest-value targets, check out the InterestingDirectories.txt file, which I blogged about here: https://danielmiessler.com/blog/the-most-interesting-disallowed-directories/.
+In other words, it's a list of potential high-value targets.
 
-If you want to see how the output is created, enter the 'Code' directory. There you can get the raw Alexa site list, the scripts that are used to download and manipulate the robots.txt files, etc.
+## The project
 
-## Credit
+So what we did is take the Alexa Top 100,000 websites, download their robots.txt files, extracted all Disallowed directories, and then performed a bunch of cleanup on them (they are a mess) to make the lists as useful as possible during web assessments.
 
-This concept is not new. The RAFT project was the first to do this, but the project is now dead and gone. And since the concept works best when it's kept up-to-date, we decided to give it a refresh in the form of RobotsDisallowed.
+### History and updates
 
-## Next steps
+- This project was initially created in 2017, and used the Alexa 100K.
+- This project was last updated in March of 2019, and used the Majestic Million top 100K.
 
-There are lots of things we want to do with this:
+## How to use the project
 
-1. Write a cleanup script that prunes the least likely hits
-2. Complete all one million sites
-3. Create individual lists for the top 10, top 100, top 1000 directories, etc. So if you're pushed for time you can use one of the condensed versions.
+You use the project by coming to the root and downloading the DisallowedDirectories files there. You can then plug them into your favorite web assessment tool/function, e.g., Burp Intruder, ZAP, etc.
 
-More ideas welcome!
+The files are broken down into Top-*N* lists, which are sorted lists based on the most common directories found. But if you are pressed for time or are looking for the highest-value targets, check out the InterestingDirectories.txt file, which I blogged about [here](https://danielmiessler.com/blog/the-most-interesting-disallowed-directories/).
 
-### Leaders
+## Credit
 
-It's harder than it looks to make the list both comprehensive and usable. People tend to have some pretty silly stuff in their robots.txt files, and many of the entries are only useful for one site.
+This concept is not new. The RAFT project was the first to do this, and we thank them for their pioneering the idea. But the project is now dead and gone, and since the idea works best when it's kept up-to-date, we decided to give it a refresh in the form of RobotsDisallowed.
 
-So we curate.
+## Shoutouts
 
-If you'd like to help out, feel free to submit issues to the repo or send pull requests.
+Thanks so much to [Jason Haddix](https://twitter.com/jhaddix)  and [Tim Tomes](https://twitter.com/lanmaster53) for talking about the project on Twitter and in their bounty/recon/hacking presentations around the world.
 
-Thanks!
+## Feedback
 
-### Credits
+If you have any ideas on what to improve, please let us know. You can email me at:
 
-It's important to us to thank people when they help out with the project.
+[email protected]
 
-- Brad Wolfe for adding the epic Bash multithreading (yeah, you read that correctly; go check the code)
+Thank you!