FAQ

Table of Contents

Quick Start

Build From Source

Features

• The ONLY open source WEB search engine.
• 64-bit architecture.
• Scalable to thousands of servers.
• Has scaled to over 12 billion web pages on over 200 servers.
• A dual quad core, with 32GB ram, and two 160GB Intel SSDs, running 8 Gigablast instances, can do about 8 qps (queries per second) on an index of 10 million pages. Drives will be close to maximum storage capacity. Doubling index size will more or less halve qps rate. (Performance metrics can be made about ten times faster but I have not got around to it yet. Drive space usage will probably remain about the same because it is already pretty efficient.)
• 1 million web pages requires 28.6GB of drive space. That includes the index, meta information and the compressed HTML of all the web pages. That is 28.6K of disk per HTML web page.
• Spider rate is around 1 page per second per core. So a dual quad core can spider and index 8 pages per second which is 691,200 pages per day.
• 4GB of RAM required per Gigablast instance. (instance = process)
• Live demo at http://www.gigablast.com/
• Written in C/C++ for optimal performance.
• Over 500,000 lines of C/C++.
• 100% custom. A single binary. The web server, database and everything else is all contained in this source code in a highly efficient manner. Makes administration and troubleshooting easier.
• Reliable. Has been tested in live production since 2002 on billions of queries on an index of over 12 billion unique web pages, 24 billion mirrored.
• Super fast and efficient. One of a small handful of search engines that have hit such big numbers. The only open source search engine that has.
• Supports all languages. Can give results in specified languages a boost over others at query time. Uses UTF-8 representation internally.
• Track record. Has been used by many clients. Has been successfully used in distributed enterprise software.
• Cached web pages with query term highlighting.
• Shows popular topics of search results (Gigabits), like a faceted search on all the possible phrases.
• Email alert monitoring. Let's you know when the system is down in all or part, or if a server is overheating, or a drive has failed or a server is consistently going out of memory, etc.
• "Synonyms" based on wiktionary data. Using query expansion method.
• Customizable "synonym" file: my-synonyms.txt
• No silly TF/IDF or Cosine. Stores position and format information (fancy bits) of each word in an indexed document. It uses this to return results that contain the query terms in close proximity rather than relying on the probabilistic tf/idf approach of other search engines.
• Complete scoring details are displayed in the search results.
• Indexes anchor text of inlinks to a web page and uses many techniques to flag pages as link spam thereby discounting their link weights.
• Demotes web pages if they are spammy.
• Can cluster results from same site.
• Duplicate removal from search results.
• Distributed web crawler/spider. Supports crawl delay and robots.txt.
• Crawler/Spider is highly programmable and URLs are binned into priority queues. Each priority queue has several throttles and knobs.
• Spider status monitor to see the urls being spidered over the whole cluster in a real-time widget.
• Complete REST/XML API for doing queries as well as adding and deleting documents in real-time.
• Automated data corruption detection, fail-over and repair based on hardware failures.
• Custom Search. (aka Custom Topic Search). Using a cgi parm like &sites=abc.com+xyz.com you can restrict the search results to a list of up to 500 subdomains.
• DMOZ integration. Run DMOZ directory. Index and search over the pages in DMOZ. Tag all pages from all sites in DMOZ for searching and displaying of DMOZ topics under each search result.
• Collections. Build tens of thousands of different collections, each treated as a separate search engine. Each can spider and be searched independently.
• Federated search over multiple Gigablast collections using syntax like &c=mycoll1+mycoll2+mycoll3+...
• Plug-ins. For indexing any file format by calling Plug-ins to convert that format to HTML. Provided binary plug-ins: pdftohtml (PDF), ppthtml (PowerPoint), antiword (MS Word), pstotext (PostScript).
• Indexes JSON and XML natively. Provides ability to search individual structured fields.
• Sorting. Sort the search results by meta tags or JSON fields that contain numbers, simply by adding something like gbsortby:price or gbrevsortby:price as a query term, assuming you have meta price tags.
• Easy Scaling. Add new servers to the hosts.conf file then click 'rebalance shards' to automatically rebalance the sharded data.
• Using &stream=1 can stream back millions of search results for a query without running out of memory.
• Makes and displays thumbnail images in the search results.
• Nested boolean queries using AND, OR, NOT operators.
• Built-in support for diffbot.com's api, which extracts various entities from web sites, like products, articles, etc. But you will need to get a free token from them for access to their API.
• Facets over meta tags or X-Paths for HTML documents.
• Facets over JSON and XML fields.
• Sort and constrain by numeric fields in JSON or XML.
• Built-in real-time profiler.
• Built-in QA tester.
• Can inject WARC and ARC archive files.

• Create the dmozparse program.
  $ make dmozparse
  
  
• Download the latest content.rdf.u8 and structure.rdf.u8 files from http://rdf.dmoz.org/rdf into the catdb/ directory onto host 0, the first host listed in the hosts.conf file.
  $ mkdir catdb
  $ cd catdb
  $ wget http://rdf.dmoz.org/rdf/content.rdf.u8.gz
  $ gunzip content.rdf.u8.gz
  $ wget http://rdf.dmoz.org/rdf/structure.rdf.u8.gz
  $ gunzip structure.rdf.u8.gz
  
  
• Execute dmozparse in its directory with the new option to generate the catdb .dat files. These .dat files are in Gigablast's special format so Gigablast can quickly get all the associated DMOZ entries given a url. Having several Missing parent for catid ... messages are normal. Ultimately, it should put two files into the catdb/ subdirectory: gbdmoz.structure.dat and gbdmoz.content.dat.
  $ cd ..
  $ ./dmozparse new
  
  
• Execute the installcat script command on host 0 to distribute the catdb files to all the hosts.
  This just does an scp from host 0 to the other hosts listed in hosts.conf.
  $ ./gb installcat
  
  
• Make sure all spiders are stopped and inactive.
  
  
• Click the catdb in the admin section of Gigablast and click "Generate Catdb" (NOT Update Catdb). This will make a huge list of catdb records and then add them to all the hosts in the network in a sharded manner.
  
  
• Once the command returns, typically in 3-4 minutes, Catdb will be ready for use and spidering. It will have created some ./catdb/catdb*.dat files which are Gigablast's database of DMOZ entry records. Any documents added that are from a site in DMOZ will show up in the search results with their appropriate DMOZ categories listed beneath. This will affect all collections.
  

Go to the catdb page and enter a url into the Lookup Category Url box and hit enter to see the associated DMOZ records for that url. So if you enter https://www.ibm.com/ you should see a few entries. It treats http as different from https, so be careful with that because http://www.ibm.com/ is not in it.

• Gigablast provides the unique ability to search the content of the pages in the DMOZ directory. But in order to search the pages in DMOZ we have to index them. You can't search what is not indexed. So execute dmozparse with the urldump -s option to create the html/gbdmoz.urls.txt.* files which contain all the URLs in DMOZ. (Excluding URLs that contained hashtags, '#'.) It will create several large files. Each file it creates is basically a VERY LARGE page of links and each link is a url in dmoz. Each of these files has a <meta name=spiderlinkslinks content=0> special Gigablast meta tag that says NOT to follow the links OF THE LINKS. So it will just spider the outlinks on this massive page and then stop. Furthermore, the massive page also has a <meta name=noindex content=1> tag that tells Gigablast to not index this massive page itself, but only spider the outlinks.
  $ ./dmozparse urldump -s
  
  
• Now tell Gigablast to index each URL listed in each gbdmoz.urls.txt.* file. Make sure you specify the collection you are using for DMOZ, in the example below it uses main. You can use the add url page to add the gbdmoz.urls.txt.* files or you can use curl (or wget) like:
  $ curl "http://127.0.0.1:8000/addurl?id=1&spiderlinks=1&c=main&u=http://127.0.0.1:8000/gbdmoz.urls.txt.0"
  $ curl "http://127.0.0.1:8000/addurl?id=1&spiderlinks=1&c=main&u=http://127.0.0.1:8000/gbdmoz.urls.txt.1"
  $ curl "http://127.0.0.1:8000/addurl?id=1&spiderlinks=1&c=main&u=http://127.0.0.1:8000/gbdmoz.urls.txt.2"
  $ curl "http://127.0.0.1:8000/addurl?id=1&spiderlinks=1&c=main&u=http://127.0.0.1:8000/gbdmoz.urls.txt.3"
  $ curl "http://127.0.0.1:8000/addurl?id=1&spiderlinks=1&c=main&u=http://127.0.0.1:8000/gbdmoz.urls.txt.4"
  $ curl "http://127.0.0.1:8000/addurl?id=1&spiderlinks=1&c=main&u=http://127.0.0.1:8000/gbdmoz.urls.txt.5"
  $ curl "http://127.0.0.1:8000/addurl?id=1&spiderlinks=1&c=main&u=http://127.0.0.1:8000/gbdmoz.urls.txt.6"
  
  
• Each gbdmoz.urls.txt.* file contains a special meta tag which instructs Gigablast to index each DMOZ URL even if there was some external error, like a DNS or TCP timeout. If the error is internal, like an Out of Memory error, then the document will, of course, not be indexed, but it should be reported in the log. This is essential for making our version of DMOZ exactly like the official version.
  
  
• Finally, ensure spiders are enabled for your collection. In the above example, main. And ALSO ensure that spiders are enabled in the Master Controls for all collections. Then the URLs you added above should be spidered and indexed. Hit reload on the Spider Queue tab to ensure you see some spider activity for your collection.

• Shutdown Gigablast.
  
• Delete catdb-saved.dat and all catdb/catdb*.dat and catdb/catdb*.map files from all hosts.
  
• Start Gigablast.
• You will have to run ./dmozparsenew new again to undelete.
  

• Dmozparse prints an error saying it could not open a file:
  Be sure you are running dmozparse in the cat directory and that the steps above have been followed correctly so that all the necessary files have been downloaded or created.
  
• Dmozparse prints an Out of Memory error:
  Some modes of dmozparse can require several hundred megabytes of system memory. Systems with insufficient memory, under heavy load, or lacking a correctly working swap may have problems running dmozparse. Attempt to free up as much memory as possible if this occcurs.
  
• How to tell if pages are being added with correct directory data:
  All pages with directory data are indexed with special terms utilizing a prefix and sufix. The prefixes are listed below and represent a specific feature under which the page was indexed. The sufix is always a numerical category ID. To search for one of these terms, simply performa a query with "prefix:sufix", i.e. "gbpcat:2" will list all pages under the Top category (or all pages in the entire directory).
  
  • gbcatid - The page is listed directly under this base category.
    
  • gbpcatid - The page is listed under this category or any child of this category.
    
  • gbicatid - The page is listed indirectly under this base category, meaning it is a page found under a site listed in the base category.
    
  • gbipcatid - The page is listed indirectly under this category, meaning it is a page found under a site listed under this category or any child of this category.
    
• Pages are not being indexed with directory data:
  First check to make sure that sites that are actually in DMOZ are those being added by the spiders. Next check to see if the sites return category information when looked up under the Catdb admin section. If they come back with directory information, the site may just need to be respidered. If the lookup does not return category information and all hosts are properly running, Catdb may need to be rebuilt from scratch.
  
• The Directory shows results but does not show sub-category listings or a page error is returned and no results are shown:
  Make sure the gbdmoz.structure.dat and structure.rdf.u8 files are in the cat directory on every host. Also be sure the current dat files were built from the current rdf.u8 files. Check the log to see if Categories was properly loaded from file at startup (grep log# Categories).
  

1091228736104 0 Gigablast Version 1.234
1091228736104 0 Allocated 435333 bytes for thread stacks.
1091228736104 0 Failed to alloc 360000 bytes.
1091228736104 0 Failed to intersect lists. Out of memory.
1091228736104 0 Too many words. Query truncated.
1091228736104 0 GET http://hohum.com/foobar.html
1091228736104 0 http://hohum.com/foobar.html ip=4.5.6.7 : Success
1091228736104 0 Skipping xxx.com, would hammer IP.