[ad_1]
Monday, September 21, 2020
Final 12 months we launched the
robots.txt parser and matcher that we use in
our manufacturing methods to the open supply world. Since then, we have seen folks construct new instruments
with it,
contribute to the
open supply library (successfully enhancing our manufacturing systems- thanks!), and launch new
language variations like golang and
rust, which make it simpler for
builders to construct new instruments.
With the intern season ending right here at Google, we needed to spotlight two new releases associated to
robots.txt that have been made attainable by two interns engaged on the Search Open Sourcing group,
Andreea Dutulescu and
Ian Dolzhanskii.
Robots.txt Specification Check
First, we’re releasing a
testing framework for robots.txt
parser builders, created by Andreea. The undertaking gives a testing software that may validate
whether or not a robots.txt parser follows the Robots Exclusion Protocol, or to what extent. At the moment
there isn’t a official and thorough approach to assess the correctness of a parser, so Andreea constructed a
software that can be utilized to create robots.txt parsers which are following the protocol.
Java robots.txt parser and matcher
Second, we’re releasing an official
Java port of the C++ robots.txt parser,
created by Ian. Java is the
third hottest programming language
on GitHub and it is extensively used at Google as nicely, so no marvel it has been essentially the most requested
language port. The parser is a 1-to-1 translation of the C++ parser by way of features and
habits, and it has been completely examined for parity towards a big corpora of robots.txt
guidelines. Groups are already planning to make use of the Java robots.txt parser in Google manufacturing
methods, and we hope that you will discover it helpful, too.
As normal, we welcome your contributions to those initiatives. In case you constructed one thing with the
C++ robots.txt parser or with these new
releases, tell us so we are able to probably assist you to unfold the phrase! In case you discovered a bug, assist
us repair it by opening a problem on GitHub or instantly contributing with a pull request. In case you
have questions or feedback about these initiatives, catch us on
Twitter!
It was our real pleasure to host Andreea and Ian, and we’re unhappy that their internship is
ending. Their contributions assist make the Web a greater place and we hope that we are able to
welcome them again to Google sooner or later.
[ad_2]
