Gigablast - api




            WEB    

            DIRECTORY    

            ADVANCED    

            ADD URL    

            SYNTAX    

            ABOUT    

            BLOG    

            FAQ    

            API    
 

ADMIN    





NOTE: All APIs support both GET and POST method. If the size of your request is more than 2K you should use POST.

NOTE: All APIs support both http and https protocols.

API by pages
/search - search results page   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3qSTRINGqueryThe query to perform. See help. See the query operators below for more info. REQUIRED
4cSTRINGcollectionSearch this collection. Use multiple collection names separated by a whitespace to search multiple collections at once. REQUIRED
5nINT32number of results per query10The number of results returned. If you want more than 1000 results you must use &stream=1 so Gigablast does not run out of memory.
6sINT32first result num0Start displaying at search result #X. Starts at 0. If you want more than 1000 results in total, you must use &stream=1 so Gigablast does not run out of memory.
7showerrorsBOOL (0 or 1)show errors0Show errors from generating search result summaries rather than just hide the docid. Useful for debugging.
8showanomaliesBOOL (0 or 1)show anomalies0Show search results that only contain the query terms in some anomalous link texts.
9scBOOL (0 or 1)site cluster0Should search results be site clustered? This limits each site to appearing at most twice in the search results. Sites are subdomains for the most part, like abc.xyz.com.
10hacrBOOL (0 or 1)hide all clustered results0Only display at most one result per site.
11drBOOL (0 or 1)dedup results0Should duplicate search results be removed? This is based on a content hash of the entire document. So documents must be exactly the same for the most part.
12pssINT32percent similar dedup summaryIf document summary (and title) are this percent similar to a document summary above it, then remove it from the search results. 100 means only to remove if exactly the same. 0 means no summary deduping. You must also supply dr=1 for this to work.
13dduBOOL (0 or 1)dedup URLs0Should we dedup URLs with case insensitivity? This is mainly to correct duplicate wiki pages.
14spellBOOL (0 or 1)do spell checking1If enabled while using the XML feed, when Gigablast finds a spelling recommendation it will be included in the XML tag. Default is 0 if using an XML feed, 1 otherwise. Will be availble again soon.
15streamCHARstream search results0Stream search results back on socket as they arrive. Useful when thousands/millions of search results are requested. Required when doing such things otherwise Gigablast could run out of memory. Only supported for JSON and XML formats, not HTML. You must use this if you want more than 1000 results.
16secsbackINT32seconds back0Limit results to pages spidered this many seconds ago. Use 0 to disable.
17sortbyCHARsort by0Use 0 to sort results by relevance, 1 to sort by most recent spider date down, and 2 to sort by oldest spidered results first.
18filetypeSTRINGfiletypeRestrict results to this filetype. Supported filetypes are pdf, doc, html xml, json, xls.
19scoresBOOL (0 or 1)get scoring infoGet scoring information for each result so you can see how each result is scored. You must explicitly request this using &scores=1 for the XML feed because it is not included by default.
20fastBOOL (0 or 1)fast results0Sacrifice some quality and result filtering for the sake of speed.
21qeBOOL (0 or 1)do query expansionIf enabled, query expansion will expand your query to include the various forms and synonyms of the query terms.
22uipSTRINGuser ipThe ip address of the searcher. We can pass back for use in the autoban technology which bans abusive IPs.
23nfINT32max number of facets to return50Max number of facets to return
24qlangSTRINGsort language preferenceDefault language to use for ranking results. Value should be any language abbreviation, for example "en" for English. Use xx to give ranking boosts to no language in particular. See the language abbreviations at the bottom of the url filters page.
25langwFLOAT32language weightUse this to override the default language weight for this collection. The default language weight can be set in the search controls and is usually something like 20.0. Which means that we multiply a result's score by 20 if from the same language as the query or the language is unknown.
26tmlINT32max title lenWhat is the maximum number of characters allowed in titles displayed in the search results?
27nsINT32number of summary excerptsHow many summary excerpts to display per search result?
28swINT32max summary line width<br> tags are inserted to keep the number of chars in the summary per line at or below this width. Also affects title. Strings without spaces that exceed this width are not split. Has no affect on xml or json feed, only works on html.
29smxcplINT32max summary excerpt lengthWhat is the maximum number of characters allowed per summary excerpt?
30dsrtINT32results to scan for gigabits generationHow many search results should we scan for gigabit (related topics) generation. Set this to zero to disable gigabits!
31iprBOOL (0 or 1)ip restriction for gigabitsShould Gigablast only get one document per IP domain and per domain for gigabits (related topics) generation?
32nrtINT32number of gigabits to show11What is the number of gigabits (related topics) displayed per query? Set to 0 to save a little CPU time.
33mtsINT32min topics scoreGigabits (related topics) with scores below this will be excluded. Scores range from 0% to over 100%.
34mdcINT32min gigabit doc count by default2How many documents must contain the gigabit (related topic) in order for it to be displayed.
35dspINT32dedup doc percent for gigabits (related topics)80If a document is this percent similar to another document with a higher score, then it will not contribute to the gigabit generation.
36mwptINT32max words per gigabit (related topic) by default6Maximum number of words a gigabit (related topic) can have. Affects xml feeds, too.
37showimagesBOOL (0 or 1)show images1Should we return or show the thumbnail images in the search results?
38usecacheCHARuse cache1Use 0 if Gigablast should not read or write from any caches at any level.
39rcacheBOOL (0 or 1)read from cache1Should we read search results from the cache?
40wcacheCHARwrite to cache1Use 0 if Gigablast should not write to any caches at any level.
41minserpdocidINT64max serp docid0Start displaying results after this score/docid pair. Used by widget to append results to end when index is volatile.
42maxserpscoreFLOAT64max serp score0Start displaying results after this score/docid pair. Used by widget to append results to end when index is volatile.
43linkSTRINGrestrict search to pages that link to this urlThe url which the pages must link to.
44sitesSTRINGrestrict results to these sitesReturned results will have URLs from these space-separated list of sites. Can have up to 200 sites. A site can include sub folders. This is allows you to build a Custom Topic Search Engine.
45ffBOOL (0 or 1)family filter0Remove objectionable results if this is enabled.
46qhBOOL (0 or 1)highlight query terms in summaries1Use to disable or enable highlighting of the query terms in the summaries.
47hqSTRINGcached page highlight queryHighlight the terms in this query instead.
48bqINT32boolean status2Can be 0 or 1 or 2. 0 means the query is NOT boolean, 1 means the query is boolean and 2 means to auto-detect.
49dtSTRINGmeta tags to displayA space-separated string of meta tag names. Do not forget to url-encode the spaces to +'s or %%20's. Gigablast will extract the contents of these specified meta tags out of the pages listed in the search results and display that content after each summary. i.e. &dt=description will display the meta description of each search result. &dt=description:32+keywords:64 will display the meta description and meta keywords of each search result and limit the fields to 32 and 64 characters respectively. When used in an XML feed the <display name="meta_tag_name">meta_tag_content</> XML tag will be used to convey each requested meta tag's content.
50nicenessINT32niceness0Can be 0 or 1. 0 is usually a faster, high-priority query, 1 is a slower, lower-priority query.
51debugCHARdebug flag0Is 1 to log debug information, 0 otherwise.
52rdcBOOL (0 or 1)return number of docs per topic1Use 1 if you want Gigablast to return the number of documents in the search results that contained each topic (gigabit).
53rdBOOL (0 or 1)return docids per topic0Use 1 if you want Gigablast to return the list of docIds from the search results that contained each topic (gigabit).
54debuggigabitsBOOL (0 or 1)debug gigabits flag0Is 1 to log gigabits debug information, 0 otherwise.
55dioBOOL (0 or 1)return docids only0Is 1 to return only docids as query results.
56adminBOOL (0 or 1)admin override1admin override
57prependSTRINGprependprepend this to the supplied query followed by a |.
58sbBOOL (0 or 1)show banned pages0show banned pages
59iccINT32include cached copy of page0Will cause a cached copy of content to be returned instead of summary.

Example XML Output (&format=xml)

<response>
	<statusCode>0</statusCode>
	<statusMsg>Success</statusMsg>
	<currentTimeUTC>1404513734</currentTimeUTC>
	<responseTimeMS>284</responseTimeMS>
	<docsInCollection>226</docsInCollection>
	<hits>193</hits>
	<moreResultsFollow>1</moreResultsFollow>
	<result>
		<imageBase64>/9j/4AAQSkZJRgABAQAAAQABA...</imageBase64>
		<imageHeight>350</imageHeight>
		<imageWidth>223</imageWidth>
		<origImageHeight>470</origImageHeight>
		<origImageWidth>300</origImageWidth>
		<title><![CDATA[U.S....]]></title>
		<sum>Department of the Interior protects America's natural resources and</sum>
		<url><![CDATA[www.doi.gov]]></url>
		<size>  64k</size>
		<docId>34111603247</docId>
		<site>www.doi.gov</site>
		<spidered>1404512549</spidered>
		<firstIndexedDateUTC>1404512549</firstIndexedDateUTC>
		<contentHash32>2680492249</contentHash32>
		<language>English</language>
	</result>
</response>

Example JSON Output (&format=json)

{ "response":{

	# This is zero on a successful query. 
	# Otherwise it will be a non-zero number 
	# indicating the error code.
	"statusCode":0,

	# Similar to above, this is "Success" 
	# on a successful query. Otherwise it 
	# will indicate an error message 
	# corresponding to the statusCode above.
	"statusMsg":"Success",

	# This is the current time in UTC in 
	# unix timestamp format (seconds since 
	# the epoch) that the server has when 
	# generating this JSON response.
	"currentTimeUTC":1404588231,

	# This is how long it took in 
	# milliseconds to generate the JSON 
	# response from reception of the request.
	"responseTimeMS":312,

	# This is how many matches were 
	# excluded from the search results 
	# because they were considered 
	# duplicates, banned, had errors 
	# generating the summary, or were from an 
	# over-represented site. To show them use 
	# the &sc &dr &pss &sb and &showerrors 
	# input parameters described above.
	"numResultsOmitted":3,

	# This is how many shards failed to 
	# return results. Gigablast gets results 
	# from multiple shards (computers) and 
	# merges them to get the final result 
	# set. Some times a shard is down or 
	# malfunctioning so it will not 
	# contribute to the results. So If this 
	# number is non-zero then you had such a 
	# shard.
	"numShardsSkipped":0,

	# This is how many shards are ideally 
	# in use by Gigablast to generate search 
	# results.
	"totalShards":159,

	# This is how many total documents are 
	# in the collection being searched.
	"docsInCollection":226,

	# This is how many of those documents 
	# matched the query.
	"hits":193,

	# This is 1 if more search results are 
	# available, otherwise it is 0.
	"moreResultsFollow":1,

	# Start of query-based information.
	"queryInfo":{

		# The entire query that was received, 
		# represented as a single string.
		"fullQuery":"test",

		# The language of the query. This is 
		# the 'preferred' language of the search 
		# results. It is reflecting the &qlang 
		# input parameter described above. Search 
		# results in this language (or an unknown 
		# language) will receive a large boost. 
		# The boost is multiplicative. The 
		# default boost size can be overridden 
		# using the &langw input parameter 
		# described above. This language 
		# abbreviation here is usually 2 letter, 
		# but can be more, like in the case of 
		# zh-cn, for example.
		"queryLanguageAbbr":"en",

		# The language of the query. Just 
		# like above but the language is spelled 
		# out. It may be multiple words.
		"queryLanguage":"English",

		# List of space separated words in 
		# the query that were ignored for the 
		# most part. Because they were common 
		# words for the query language they are 
		# in.
		"ignoredWords":"to the",

		# There is a maximum limit placed on 
		# the number of query terms we search on 
		# to keep things fast. This can be 
		# changed in the search controls.
		"queryNumTermsTotal":52,
		"queryNumTermsUsed":20,
		"queryWasTruncated":1,

		# The start of the terms array. Each 
		# query is broken down into a list of 
		# terms. Each term is described here.
		"terms":[

			# The first query term in the JSON 
			# terms array.
			{

			# The term number, starting at 0.
			"termNum":0,

			# The term as a string.
			"termStr":"test",

			# The term frequency. An estimate of 
			# how many pages in the collection 
			# contain the term. Helps us weight terms 
			# by popularity when scoring the results.
			"termFreq":425239458,

			# A 48-bit hash of the term. Used to 
			# represent the term in the index.
			"termHash48":67259736306430,

			# A 64-bit hash of the term.
			"termHash64":9448336835959712000,

			# If the term has a field, like the 
			# term title:cat, then what is the hash 
			# of the field. In this example it would 
			# be the hash of 'title'. But for the 
			# query 'test' there is no field so it is 
			# 0.
			"prefixHash64":0

			},

			# The second query term in the JSON 
			# terms array.
			{

			"termNum":1,
			"termStr":"tested",

			# The language the term is from, in 
			# the case of query expansion on the 
			# original query term. Gigablast tries to 
			# find multiple forms of the word that 
			# have the same essential meaning. It 
			# uses the specified query language 
			# (&qlang), however, if a query term is 
			# from a different language, then that 
			# language will be implied for query 
			# expansion.
			"termLang":"en",

			# The query term that this term is a 
			# form of.
			"synonymOf":"test",

			"termFreq":73338909,
			"termHash48":66292713121321,
			"termHash64":9448336835959712000,
			"prefixHash64":0
			},

			...

		# End of the JSON terms array.
		]

	# End of the queryInfo JSON structure.
	},

	# The start of the gigabits array. 
	# Each gigabit is mined from the content 
	# of the search results. The top N 
	# results are mined, and you can control 
	# N with the &dsrt input parameter 
	# described above.
	"gigabits":[

		# The first gigabit in the array.
		{

		# The gigabit as a string in utf8.
		"term":"Membership",

		# The numeric score of the gigabit.
		"score":240,

		# The popularity ranking of the 
		# gigabit. Out of 10000 random documents, 
		# how many documents contain it?
		"minPop":480,

		# The gigabit in the context of a 
		# document.
		"instance":{

			# A sentence, if it exists, from one 
			# of the search results which also 
			# contains the gigabit and as many 
			# significant query terms as possible. In 
			# UTF-8.
			"sentence":"Get a free Tested Premium Membership here!",

			# The url that contained that 
			# sentence. Always starts with http.
			"url":"http://www.tested.com/",

			# The domain of that url.
			"domain":"tested.com"
		}

		# End of the first gigabit
		},

		...

	# End of the JSON gigabits array.
	],

	# Start of the facets array, if any.
	"facets":[

		# The first facet in the array.
		{
			# The field you are faceting over
			"field":"Company",

			# How many documents in the 
			# collection had this particular field? 
			# 64-bit integer.
			"totalDocsWithField":148553,

			# How many documents in the 
			# collection had this particular field 
			# with the same value as the value line 
			# directly below? This should always be 
			# less than or equal to the 
			# totalDocsWithField count. 64-bit 
			# integer.
			"totalDocsWithFieldAndValue":44184,

			# The value of the field in the case 
			# of this facet. Can be a string or an 
			# integer or a float, depending on the 
			# type described in the gbfacet query 
			# term. i.e. gbfacetstr, gbfacetint or 
			# gbfacetfloat.
			"value":"Widgets, Inc.",

			# Should be the same as 
			# totalDocsWithFieldAndValue, above. 
			# 64-bit integer.
			"docCount":44184

		# End of the first facet in the array.
		}

	# End of the facets array.
	],

	# Start of the JSON array of 
	# individual search results.
	"results":[

		# The first result in the array.
		{

		# The title of the result. In UTF-8.
		"title":"This is the title.",

		# A DMOZ entry. One result can have 
		# multiple DMOZ entries.
		"dmozEntry":{

			# The DMOZ category ID.
			"dmozCatId":374449,

			# The DMOZ direct category ID.
			"directCatId":1,

			# The DMOZ category as a UTF-8 
			# string.
			"dmozCatStr":"Top: Computers: Security: Malicious 
			 Software: Viruses: Detection and Removal Tools: 
			 Reviews",

			# What title some DMOZ editor gave 
			# to this url.
			"dmozTitle":"The DMOZ Title",

			# What summary some DMOZ editor gave 
			# to this url.
			"dmozSum":"A great web page.",

			# The DMOZ anchor text, if any.
			"dmozAnchor":"",

		# End DMOZ entry.
		},

		# The content type of the url. Can be 
		# html, pdf, text, xml, json, doc, xls or 
		# ps.
		"contentType":"html",

		# The summary excerpt of the result. 
		# In UTF-8.
		"sum":"Department of the Interior protects America's natural resources.",

		# The url of the result. If it starts 
		# with http:// then that is omitted. Also 
		# omits the trailing / if the urls is 
		# just a domain or subdomain on the root 
		# path.
		"url":"www.doi.gov",

		# The hopcount of the url. The 
		# minimum number of links we would have 
		# to click to get to it from a root url. 
		# If this is 0 that means the url is a 
		# root url, like http://www.root.com/.
		"hopCount":0,

		# The size of the result's content. 
		# Always in kilobytes. k stands for 
		# kilobytes. Could be a floating point 
		# number or and integer.
		"size":"  64k",

		# The exact size of the result's 
		# content in bytes.
		"sizeInBytes":64560,

		# The unique document identifier of 
		# the result. Used for getting the cached 
		# content of the url.
		"docId":34111603247,

		# The site the result comes from. 
		# Usually a subdomain, but can also 
		# include part of the URL path, like, 
		# abc.com/users/brad/. A site is a set of 
		# web pages controlled by the same 
		# entity.
		"site":"www.doi.gov",

		# The time the url was last INDEXED. 
		# If there was an error or the url's 
		# content was unchanged since last 
		# download, then this time will remain 
		# unchanged because the document is not 
		# reindexed in those cases. Time is in 
		# unix timestamp format and is in UTC.
		"spidered":1404512549,

		# The first time the url was 
		# successfully INDEXED. Time is in unix 
		# timestamp format and is in UTC.
		"firstIndexedDateUTC":1404512549,

		# A 32-bit hash of the url's content. 
		# It is used to determine if the content 
		# changes the next time we download it.
		"contentHash32":2680492249,

		# The dominant language that the 
		# url's content is in. The language name 
		# is spelled out in its entirety.
		"language":"English"

		# A convenient abbreviation of the 
		# above language. Most are two 
		# characters, but some, like zh-cn, are 
		# more.
		"langAbbr":"en"

		# If the result has an associated 
		# image then the image thumbnail is 
		# encoded in base64 format here. It is a 
		# jpg image.
		"imageBase64":"/9j/4AAQSkZJR...",

		# If the result has an associated 
		# image then what is its height and width 
		# of the above jpg thumbnail image in 
		# pixels?
		"imageHeight":223,
		"imageWidth":350,

		# If the result has an associated 
		# image then what are the dimensions of 
		# the original image in pixels?
		"origImageHeight":300,
		"origImageWidth":470

		# End of the first result.
		},

		...

	# End of the JSON results array.
	]

# End of the response.
}

}



/get - gets cached web page   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3dINT64docId0The docid of the cached page to view. REQUIRED
4urlSTRINGurlInstead of specifying a docid, you can get the cached webpage by url as well. REQUIRED
5cSTRINGcollectionGet the cached page from this collection. REQUIRED
6stripINT32strip0Is 1 or 2 two strip various tags from the cached content.
7ihBOOL (0 or 1)include header1Is 1 to include the Gigablast header at the top of the cached page, 0 to exclude the header.
8qSTRINGqueryHighlight this query in the page.

Example XML Output (&format=xml)
<response>
	<statusCode>0</statusCode>
	<statusMsg>Success</statusMsg>
	<url><![CDATA[http://www.doi.gov/]]></url>
	<docId>34111603247</docId>
	<cachedTimeUTC>1404512549</cachedTimeUTC>
	<cachedTimeStr>Jul 04, 2014 UTC</cachedTimeStr>
	<content><![CDATA[<html><title>Some web page title</title><head>My first web page</head></html>]]></content>
</response>

Example JSON Output (&format=json)
{ "response":{
	"statusCode":0,
	"statusMsg":"Success",
	"url":"http://www.doi.gov/",
	"docId":34111603247,
	"cachedTimeUTC":1404512549,
	"cachedTimeStr":"Jul 04, 2014 UTC",
	"content":"<html><title>Some web page title</title><head>My first web page</head></html>"
}
}



/admin/status - basic status   [ show parms in xml or json ]   [ show status in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionUse this collection. REQUIRED



/admin/collectionpasswords - passwords   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.



/admin/hosts - hosts status   [ show parms in xml or json ]   [ show status in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.



/admin/master - master controls   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3qrtsSTRINGquery routesdefault->127.0.0.1:7000Match a subtring in the HTTP request to the first substring before the -> below and route it, round robin style, to one host in the following comma-separated list of IP:UDP_PORT. Put an * immediately after the -> to mark the host as unpingable; just a hack for older clusters that use a different ping packet format.
4seBOOL (0 or 1)spidering enabled1Controls all spidering for all collections
5dsdcBOOL (0 or 1)do spiderdb decimation0If enabled limits to MAXURLSPERFIRSTIP urls in spiderdb per firstip. Currently 100M.
6injenBOOL (0 or 1)injections enabled1Controls injecting for all collections
7qryenBOOL (0 or 1)querying enabled1Controls querying for all collections
8dospellcheckingBOOL (0 or 1)do spell checking0Spell check all queries?
9usesortdbBOOL (0 or 1)use sortdb0A temp parm for debugging.
10rraBOOL (0 or 1)return results even if a shard is down1If you turn this off then Gigablast will return an error message if a shard was down and did not return results for a query. The XML and JSON feed let's you know when a shard is down and will give you the results back any way, but if you would rather have just and error message and no results, then set then set this to 'NO'.
11maxmemINT64max mem8000000000Mem available to this process. May be exceeded due to fragmentation.
12mtspINT32max total spiders100What is the maximum number of web pages the spider is allowed to download simultaneously for ALL collections PER HOST? Caution: raising this too high could result in some Out of Memory (OOM) errors. The hard limit is currently 300. Each collection has its own limit in the spider controls that you may have to increase as well.
13msptINT32max spiders per token30The maximum number of simultaneous spider requests outstanding per host.
14aeBOOL (0 or 1)add url enabled1Can people use the add url interface to add urls to the index?
15ucpBOOL (0 or 1)use collection passwords0Should collections have individual password settings so different users can administrer different collections? If not the only the master passwords and IPs will be able to administer any collection.
16acuBOOL (0 or 1)allow cloud users0Can guest users create and administer a collection? Limit: 1 collection per IP address. This is mainly for doing demos on the gigablast.com domain.
17asfINT32auto save frequency5Save data in memory to disk after this many minutes have passed without the data having been dumped or saved to disk. I've seen this block over 50 seconds on a busy server before that used SSDs, so be aware. We now force this to a minimum of half a day, 60*12 to avoid collection configuration data loss.
18mhsINT32max http sockets300Maximum sockets available to serve incoming HTTP requests. Too many outstanding requests will increase query latency. Excess requests will simply have their sockets closed.
19mssINT32max https sockets100Maximum sockets available to serve incoming HTTPS requests. Like max http sockets, but for secure sockets.
20ttosecsINT32tcp time out60How many seconds to wait before timing out a TCP download?
21ttosecsproxyINT32tcp time out (proxy)80How many seconds to wait before timing out a TCP download over a proxy?
22suaSTRINGspider user agentGigablastOpenSource/1.0Identification seen by web servers when the Gigablast spider downloads their web pages. It is polite to insert a contact email address here so webmasters that experience problems from the Gigablast spider have somewhere to vent.
23jsUNARY CMD (set to 1)saveSaves in-memory data for ALL hosts. Does Not exit. If running from a proxy, then just saves the gb.conf data for that proxy.
24saveUNARY CMD (set to 1)save & exitSaves the data and exits for ALL hosts. If running on a proxy, just saves and exits on that proxy.
25rebalanceUNARY CMD (set to 1)rebalance shardsTell all hosts to scan all records in all databases, and move records to the shard they belong to. You only need to run this if Gigablast tells you to, when you are changing hosts.conf to add or remove more nodes/hosts.
26dumpUNARY CMD (set to 1)dump to diskFlushes all records in memory to the disk on all hosts.
27clrkrnerrUNARY CMD (set to 1)clear errorsClears the kernel error messages, the out of memory and the core-dump signals on the hosts page. (The x and O letter signals.)
28fingerprintUNARY CMD (set to 1)record user agent fingerprintRecord the user agent from your browser along with the standard headers that this user agent sends with its requests
29meenBOOL (0 or 1)merges enabled1Disable at startup to make startup faster.
30tmaINT32tight merge after this many days0If oldest file is this many days old then do a tight merge. Used by diffbot to realize spider status doc deletes faster. Use 0 to not do this. Will only do the tight merge if a regular merge is triggered from the number of files being too large.
31afgdwdBOOL (0 or 1)ask for gzipped docs when downloading0If this is true, gb will send Accept-Encoding: gzip to web servers when doing http downloads. It does have a tendency to cause out-of-memory errors when you enable this, so until that is fixed better, it's probably a good idea to leave this disabled.
32abBOOL (0 or 1)autoban IPs which violate the queries per day quotas0Keep track of ips which do queries, disallow non-customers from hitting us too hard.
33nfqpdINT32free queries per day 1024Non-customers get this many queries per day beforebeing autobanned
34nfqpmINT32free queries per minute 30Non-customers get this many queries per minute beforebeing autobanned
35ubfBOOL (0 or 1)use bot fence0Use ajax to load the search results to keep bots from slamming gb with queries.
36bfswSTRINGsecret word for bot fencerxiwdUse ajax to set the value of this word as a cgi parm in the url. TODO: make this automatic again.
37mhdmsINT32max heartbeat delay in milliseconds0If a heartbeat is delayed this many milliseconds dump a core so we can see where the CPU was. Logs 'db: missed heartbeat by %ld ms'. Use 0 or less to disable.
38mdchINT32max delay before logging a callback or handler-1If a call to a message callback or message handler in the udp server takes more than this many milliseconds, then log it. Logs 'udp: Took %ld ms to call callback for msgType=0x%hhx niceness=%d'. Use -1 or less to disable the logging.
39dhtINT32dead host timeout6000Consider a host in the Gigablast network to be dead if it does not respond to successive pings for this number of seconds. Gigablast does not send requests to dead hosts. Outstanding requests may be re-routed to a twin.
40psmsINT32ping spacer100Wait this many milliseconds before pinging the next host. Each host pings all other hosts in the network.
41errstroneSTRINGerror string 1I/O errorLook for this string in the kernel buffer for sending email alert. Useful for detecting some strange hard drive failures that really slow performance.
42errstrtwoSTRINGerror string 2Look for this string in the kernel buffer for sending email alert. Useful for detecting some strange hard drive failures that really slow performance.
43errstrthreeSTRINGerror string 3Look for this string in the kernel buffer for sending email alert. Useful for detecting some strange hard drive failures that really slow performance.
44dpcspINT64posdb disk cache size for small termlists30000000How much file cache size to use in bytes for termlists smaller than 1MB? Posdb is the index.
45balloonINT64posdb disk cache size for large termlists100000000How much file cache size to use in bytes for termlists greater than or equal to 1MB? Posdb is the index.
46dpcstINT64tagdb disk cache size30000000How much file cache size to use in bytes? Tagdb is consulted at spider time and query time to determine if a url or outlink is banned or what its siterank is, etc.
47dpcscINT64clusterdb disk cache size30000000How much file cache size to use in bytes? Gigablast does a lookup in clusterdb for each search result at query time to get its site information for site clustering. If you disable site clustering in the search controls then clusterdb will not be consulted.
48dpcsxINT64titledb disk cache size30000000How much file cache size to use in bytes? Titledb holds the cached web pages, compressed. Gigablast consults it to generate a summary for a search result, or to see if a url Gigablast is spidering is already in the index.
49dpcsyINT64spiderdb disk cache size30000000How much file cache size to use in bytes? Titledb holds the cached web pages, compressed. Gigablast consults it to generate a summary for a search result, or to see if a url Gigablast is spidering is already in the index.
50dnsmcmINT32dns cache size30000000Bytes to use for caching dns replies per host.
51wlmcmINT32winner list cache size30000000Bytes to use for caching urls to spider for an IP. Saves CPU to getUrlFilterNum().
52wlcmaINT32winner list cache max age7200Age in seconds that entries will expire. Keep low if you want freshness as a url might not respider when it is scheduled.
53rtmcmINT32robots.txt cache size50000000Bytes to use for caching robots.txt files for spidering per host.
54hpmcmINT32html pages cache size10000000Bytes to use for caching html pages for spidering per host.
55serpmcmINT32search results cache size10000000Bytes to use for caching search result pages per host.
56srcmaINT32search results cache max age10800How many seconds should we cache a search results page for?
57srtcpBOOL (0 or 1)send requests to compression proxy0If this is true, gb will route download requests for web pages to proxies in hosts.conf. Proxies will download and compress docs before sending back.
58pdnsIPdns 08.8.8.8IP address of the primary DNS server. Assumes UDP port 53. REQUIRED FOR SPIDERING! Use Google's public DNS 8.8.8.8 as default.
59sdnsIPdns 18.8.4.4IP address of the secondary DNS server. Assumes UDP port 53. Will be accessed in conjunction with the primary dns, so make sure this is always up. An ip of 0 means disabled. Google's secondary public DNS is 8.8.4.4.
60sdnsaIPdns 20.0.0.0All hosts send to these DNSes based on hash of the subdomain to try to split DNS load evenly.
61sdnsbIPdns 30.0.0.0
62sdnscIPdns 40.0.0.0
63sdnsdIPdns 50.0.0.0
64sdnseIPdns 60.0.0.0
65sdnsfIPdns 70.0.0.0
66sdnsgIPdns 80.0.0.0
67sdnshIPdns 90.0.0.0
68sdnsiIPdns 100.0.0.0
69sdnsjIPdns 110.0.0.0
70sdnskIPdns 120.0.0.0
71sdnslIPdns 130.0.0.0
72sdnsmIPdns 140.0.0.0
73sdnsnIPdns 150.0.0.0
74utBOOL (0 or 1)use threads1If enabled, Gigablast will use threads.
75utfdBOOL (0 or 1)use threads for disk1If enabled, Gigablast will use threads for disk ops. Now that Gigablast uses pthreads more effectively, leave this enabled for optimal performance in all cases.
76utfioBOOL (0 or 1)use threads for intersects and merges1If enabled, Gigablast will use threads for these ops. Default is now on in the event you have simultaneous queries so one query does not hold back the other. There seems to be a bug so leave this ON for now.
77utfscBOOL (0 or 1)use threads for system calls1Gigablast does not make too many system calls so leave this on in case the system call is slow.
78mctINT32max cpu threads12Maximum number of threads to use per Gigablast process for intersecting docid lists.
79fwBOOL (0 or 1)flush disk writes1If enabled then all writes will be flushed to disk. If not enabled, then gb uses the Linux disk write cache and any hardware write cache. We seem to get better query performance when you enable this, but your disk writing operations will take longer. If this is disabled then the Linux page cache and disk write scheduler can fuck up your performance quite severely. The system can also run out of memory very easily and kill gb processes because it uses all the memory to cache writes.
80mbtfINT32min bytes to flush2000000Only call flush after we've written this many bytes on a particular file descriptor. A zero here means to flush after every disk write. Reducing flushing improves write performance. But flushing too little impacts disk read performance for queries as the kernel or hardware may only flush when its buffer is full, thereby causing spikey contention. Not flushing at all can cause the system to run out of memory and the OOM (out of memory) killer to kill gb processes.
81vwlBOOL (0 or 1)verify written lists0Ensure lists being written to disk are not corrupt. That title recs appear valid, etc. Helps isolate sources of corruption. Used for debugging.
82cmbdBOOL (0 or 1)check memory before dumping0Check the in-memory structures before dumping to disk when they are full. This will SLOW DOWN your query response time during dumps, maybe even freezing things up for 10 seconds while checking. But this will detect BAD RAM DIMMS quite well.
83vdwBOOL (0 or 1)verify disk writes0Read what was written in a verification step. Decreases performance, but may help fight disk corruption mostly on Maxtors and Western Digitals.
84smdtINT32max spider read threads20Maximum number of threads to use per Gigablast process for accessing the disk for index-building purposes. Keep low to reduce impact on query response time. Increase for fast disks or when preferring build speed over lower query latencies
85sdtBOOL (0 or 1)separate disk reads1If enabled then we will not launch a low priority disk read or write while a high priority is outstanding. Help improve query response time at the expense of spider performance.
86mbsINT32merge buf size500000Read and write this many bytes at a time when merging files. Smaller values are kinder to query performance, but the merge takes longer. Use at least 1000000 for fast merging.
87usdbBOOL (0 or 1)use statsdb1Archive system statistics information in Statsdb.
88auhsBOOL (0 or 1)always use https1Redirect all http connections to https
89asiBOOL (0 or 1)allowing spidering of local IPs0For security we default to no here, assuming you want to index the web and not your intranet. Indexing documents with local IPs allows one to accidentally index private documents from an internal web server.



/admin/search - search controls   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionUse this collection. REQUIRED
4mtbtrINT64max bytes to read90000000Maximum number of bytes to read for a termlist per shard from a single file.
5msrpqINT32max search results per query100What is the limit to the total number of returned search results per query?
6msrINT32max search results in total200What is the maximum total number of returned search results.
7msrfpcINT32max search results in total for paying clients1000What is the limit to the total number of returned search results for clients.
8langweightFLOAT32language weight240.000000Defalt language weight if document matches query language, which can be provided using &qlang= or it the default can be set in the search controls. Use this to give results that match the specified the speicified &qlang higher ranking, or docs whose language is unknown. Can be override with &langw in the query url. Set to -1 to only return results in the query language, given by &qlang= or the default query language specified in the search controls. Setting to -1 is prefered over using the gblang: query operator because it is faster.
9mqtINT32max query terms999999Do not allow more than this many query terms. Helps prevent big queries from resource hogging.
10spellBOOL (0 or 1)do spell checking by default1If enabled while using the XML feed, when Gigablast finds a spelling recommendation it will be included in the XML tag. Default is 0 if using an XML feed, 1 otherwise.
11scoresBOOL (0 or 1)get scoring info by default1Get scoring information for each result so you can see how each result is scored. You must explicitly request this using &scores=1 for the XML feed because it is not included by default.
12qeBOOL (0 or 1)do query expansion by default1If enabled, query expansion will expand your query to include the various forms and synonyms of the query terms.
13qhBOOL (0 or 1)highlight query terms in summaries by default1Use to disable or enable highlighting of the query terms in the summaries.
14tmlINT32max title len80What is the maximum number of characters allowed in titles displayed in the search results?
15scdBOOL (0 or 1)site cluster by default0Should search results be site clustered? This limits each site to appearing at most twice in the search results. Sites are subdomains for the most part, like abc.xyz.com.
16hacrBOOL (0 or 1)hide all clustered results0Only display at most one result per site.
17drdBOOL (0 or 1)dedup results by default1Should duplicate search results be removed? This is based on a content hash of the entire document. So documents must be exactly the same for the most part.
18stgdblBOOL (0 or 1)do tagdb lookups for queries1For each search result a tagdb lookup is made, usually across the network on distributed clusters, to see if the URL's site has been manually banned in tagdb. If you don't manually ban sites then turn this off for extra speed.
19psdsINT32percent similar dedup summary default value90If document summary (and title) are this percent similar to a document summary above it, then remove it from the search results. 100 means only to remove if exactly the same. 0 means no summary deduping.
20msldINT32number of lines to use in summary to dedup4Sets the number of lines to generate for summary deduping. This is to help the deduping process not throw out valid summaries when normally displayed summaries are smaller values. Requires percent similar dedup summary to be non-zero.
21dduBOOL (0 or 1)dedup URLs by default0Should we dedup URLs with case insensitivity? This is mainly to correct duplicate wiki pages.
22defqlangSTRINGsort language preference defaultenDefault language to use for ranking results. Value should be any language abbreviation, for example "en" for English. Use xx to give ranking boosts to no language in particular. See the language abbreviations at the bottom of the url filters page.
23qcountrySTRINGsort country preference defaultusDefault country to use for ranking results. Value should be any country code abbreviation, for example "us" for United States. This is currently not working.
24smlINT32max summary len512What is the maximum number of characters displayed in a summary for a search result?
25smnlINT32max summary excerpts4What is the maximum number of excerpts displayed in the summary of a search result?
26smxcplINT32max summary excerpt length90What is the maximum number of characters allowed per summary excerpt?
27smwINT32max summary line width by default80<br> tags are inserted to keep the number of chars in the summary per line at or below this width. Also affects title. Strings without spaces that exceed this width are not split. Has no affect on xml or json feed, only works on html.
28clmfsINT32bytes of doc to scan for summary generation70000Truncating this will miss out on good summaries, but performance will increase.
29sfhtSTRINGfront highlight tagFront html tag used for highlightig query terms in the summaries displated in the search results.
30sbhtSTRINGback highlight tagFront html tag used for highlightig query terms in the summaries displated in the search results.
31dsrtINT32results to scan for gigabits generation by default30How many search results should we scan for gigabit (related topics) generation. Set this to zero to disable gigabits generation by default.
32iprBOOL (0 or 1)ip restriction for gigabits by default0Should Gigablast only get one document per IP domain and per domain for gigabits (related topics) generation?
33rotBOOL (0 or 1)remove overlapping topics1Should Gigablast remove overlapping topics (gigabits)?
34nrtINT32number of gigabits to show by default11What is the number of related topics (gigabits) displayed per query? Set to 0 to save CPU time.
35mtsINT32min gigabit score by default5Gigabits (related topics) with scores below this will be excluded. Scores range from 0% to over 100%.
36mdcINT32min gigabit doc count by default2How many documents must contain the gigabit (related topic) in order for it to be displayed.
37dspINT32dedup doc percent for gigabits (related topics)80If a document is this percent similar to another document with a higher score, then it will not contribute to the gigabit generation.
38mwptINT32max words per gigabit (related topic) by default6Maximum number of words a gigabit (related topic) can have. Affects xml feeds, too.
39tmssINT32gigabit max sample size4096Max chars to sample from each doc for gigabits (related topics).
40ddcBOOL (0 or 1)display dmoz categories in results1If enabled, results in dmoz will display their categories on the results page.
41didcBOOL (0 or 1)display indirect dmoz categories in results0If enabled, results in dmoz will display their indirect categories on the results page.
42dsclBOOL (0 or 1)display Search Category link to query category of result0If enabled, a link will appear next to each category on each result allowing the user to perform their query on that entire category.
43udfuBOOL (0 or 1)use dmoz for untitled1Yes to use DMOZ given title when a page is untitled but is in DMOZ.
44udsmBOOL (0 or 1)show dmoz summaries1Yes to always show DMOZ summaries with search results that are in DMOZ.
45sacotBOOL (0 or 1)show adult category on top0Yes to display the Adult category in the Top category
46hpSTRINGhome pageHtml to display for the home page. Leave empty for default home page. Use %N for total number of pages indexed. Use %n for number of pages indexed for the current collection. Use %c to insert the current collection name. Use %q to display the query in a text box. Use %t to display the directory TOP. Example to paste into textbox:
<html><title>My Gigablast Search Engine</title><script> function x(){document.f.q.focus();} </script><body onload="x()"><br><br><center><a href=/><img border=0 width=500 height=122 src=/logo-med.jpg></a><br><br><b>My Search Engine</b><br><br><form method=get action=/search name=f><input type=hidden name=c value="%c"><input name=q type=text size=60 value="">&nbsp;<input type="submit" value="Search"></form><br><center>Searching the <b>%c</b> collection of %n documents.</center><br></body></html>
47hhSTRINGhtml headHtml to display before the search results. Leave empty for default. Convenient for changing colors and displaying logos. Use the variable, %q, to represent the query to display in a text box. Use %e to print the url encoded query. Use %S to print sort by date or relevance link. Use %L to display the logo. Use %R to display radio buttons for site search. Use %F to begin the form. and use %H to insert hidden text boxes of parameters like the current search result page number. BOTH %F and %H are necessary for the html head, but do not duplicate them in the html tail. Use %f to display the family filter radio buttons. Example to paste into textbox:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <title>My Gigablast Search Results</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body%l> %F<table cellpadding="2" cellspacing="0" border="0"> <tr> <td valign=top>%L</td> <td valign=top> <nobr> <input type="text" name="q" size="60" value="%q"> %D<input type="submit" value="Blast It!" border="0"> </nobr> <br>%f %R </tr> </table> %H
48htSTRINGhtml tailHtml to display after the search results. Leave empty for default. Convenient for changing colors and displaying logos. Use the variable, %q, to represent the query to display in a text box. Use %e to print the url encoded query. Use %S to print sort by date or relevance link. Use %L to display the logo. Use %R to display radio buttons for site search. Use %F to begin the form. and use %H to insert hidden text boxes of parameters like the current search result page number. BOTH %F and %H are necessary for the html head, but do not duplicate them in the html tail. Use %f to display the family filter radio buttons. Example to paste into textbox:
<br> <table cellpadding=2 cellspacing=0 border=0> <tr><td></td> <td>%s</td> </tr> </table> Try your search on <a href=http://www.google.com/search?q=%e>google</a> &nbsp; <a href=http://search.yahoo.com/bin/search?p=%e>yahoo</a> &nbsp; <a href=http://search.dmoz.org/cgi-bin/search?search=%e>dmoz</a> &nbsp; </font></body>
49diffbotOutputFormatBOOL (0 or 1)imitate the output format of crawlbot0
50outputJsonResultsDirectlyBOOL (0 or 1)output json results inline0



/admin/spider - spider controls   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionUse this collection. REQUIRED
4cseBOOL (0 or 1)spidering enabled1Controls just the spiders for this collection.
5sitelisttagsBOOL (0 or 1)index site tags0Index the tags from the sitelist as metadata
6sitelistSTRINGsite listList of sites to spider, one per line. See example site list below. Gigablast uses the insitelist directive on the url filters page to make sure that the spider only indexes urls that match the site patterns you specify here, other than urls you add individually via the add urls or inject url tools. Limit list to 300MB. If you have a lot of INDIVIDUAL urls to add then consider using the addurl interface.
7restartUNARY CMD (set to 1)restart collectionRemove all documents from the collection and re-add seed urls from site list.
8pmergeUNARY CMD (set to 1)tight merge posdbMerges all outstanding posdb (index) files.
9tmergeUNARY CMD (set to 1)tight merge titledbMerges all outstanding titledb (web page cache) files.
10spmergeUNARY CMD (set to 1)tight merge spiderdbMerges all outstanding spiderdb files.
11mnsINT32max spiders300What is the maximum number of web pages the spider is allowed to download simultaneously PER HOST for THIS collection? The maximum number of spiders over all collections is controlled in the master controls.
12ssfINT32spiderdb scan frequency60Rescan spiderdb every X minutes. This will ensure that urls are spidered somewhat promptly. We may not spider some high priority urls promptly because lower priority urls from the same IP address are occupying the schedule. This will help fix such situations by rebuilding the schedule from scratch every X minutes. The minimum value is 5 minutes, but the recommended minimum is probably 15 minutes.
13sdmsINT32spider delay in milliseconds0make each spider wait this many milliseconds before getting the ip and downloading the page.
14obeyRobotsBOOL (0 or 1)obey robots.txt1If this is true Gigablast will respect the robots.txt convention and rel no follow meta tags.
15usecloakingBOOL (0 or 1)use cloaking1Pretend our user-agent is googlebot for the purposes of parsing robots.txt. Does not actually set our user agent to googlebot.
16obeyRelNoFollowBOOL (0 or 1)obey rel no follow links1If this is true Gigablast will respect the rel no follow link attribute.
17mrcaINT32max robots.txt cache age86400How many seconds to cache a robots.txt file for. 86400 is 1 day. 0 means Gigablast will not read from the cache at all and will download the robots.txt before every page if robots.txt use is enabled above. However, if this is 0 then Gigablast will still store robots.txt files in the cache.
18cuaSTRINGuser agentIf empty then gigablast uses the default user agent specified in the master controls, otherwise, it uses this user agent for this collection only.
19useProxiesBOOL (0 or 1)always use spider proxies0If this is true Gigablast will ALWAYS use the proxies listed on the proxies page for spidering for this collection.
20automaticallyuseproxiesBOOL (0 or 1)automatically use spider proxies0Use the spider proxies listed on the proxies page if gb detects that a webserver is throttling the spiders. This way we can learn the webserver's spidering policy so that our spiders can be more polite. If no proxies are listed on the proxies page then this parameter will have no effect. When this is enabled gb will also use proxies for the sites listed in the sites to always use proxies text box on the proxies controls page which applies to all collections. It's basically a list of the sites we know are hard to spider without proxies.
21useeliteproxiesBOOL (0 or 1)use elite spider proxies0Use the elite spider proxies instead of the regular proxies.
22automaticallybackoffBOOL (0 or 1)automatically back off0Set the crawl delay to 5 seconds if gb detects that an IP is throttling or banning gigabot from crawling it. The crawl delay just applies to that IP. Such throttling will be logged.
23downloadiframesBOOL (0 or 1)expand iframes in spidered pages1Include iframes in the content of spidered pages.
24usetimeaxisBOOL (0 or 1)use time axis0If this is true Gigablast will index the same url multiple times if its content varies over time, rather than overwriting the older version in the index. Useful for archive web pages as they change over time.
25indexwarcsBOOL (0 or 1)index warc or arc files0If this is true Gigablast will index .warc and .arc files by injecting the pages contained in them as if they were spidered with the content in the .warc or .arc file. The spidered time will be taken from the archive file as well.
26dmtINT32daily merge time-1Do a tight merge on posdb and titledb at this time every day. This is expressed in MINUTES past midnight UTC. UTC is 5 hours ahead of EST and 7 hours ahead of MST. Leave this as -1 to NOT perform a daily merge. To merge at midnight EST use 60*5=300 and midnight MST use 60*7=420.
27dmdlSTRINGdaily merge days0Comma separated list of days to merge on. Use 0 for Sunday, 1 for Monday, ... 6 for Saturday. Leaving this parmaeter empty or without any numbers will make the daily merge happen every day
28dttBOOL (0 or 1)turing test enabled0If this is true, users will have to pass a simple Turing test to add a url. This prevents automated url submission.
29mauINT32max add urls0Maximum number of urls that can be submitted via the addurl interface, per IP domain, per 24 hour period. A value less than or equal to zero implies no limit.
30deBOOL (0 or 1)deduping enabled0When enabled, the spider will discard web pages which are identical to other web pages that are already in the index. It most likely has to hit disk to do these checks so it does cause some slow down. Only use it if you need it.
31dewBOOL (0 or 1)deduping enabled for www1When enabled, the spider will discard web pages which, when a www is prepended to the page's url, result in a url already in the index. Also, if the url exists in the index with https:// instead of http:// or with https://www.
32dcepBOOL (0 or 1)detect custom error pages1Detect and do not index pages which have a 200 status code, but are likely to be error pages.
33usrBOOL (0 or 1)use simplified redirects1If this is true, the spider, when a url redirects to a "simpler" url, will add that simpler url into the spider queue and abandon the spidering of the current url.
34useCanonicalBOOL (0 or 1)use canonical redirects1If page has a on it then treat it as a redirect, add it to spiderdb for spidering and abandon the indexing of the current url.
35uimsBOOL (0 or 1)use ifModifiedSince0If this is true, the spider, when updating a web page that is already in the index, will not even download the whole page if it hasn't been updated since the last time Gigablast spidered it. This is primarily a bandwidth saving feature. It relies on the remote webserver's returned Last-Modified-Since field being accurate.
36mtotiINT32max term occurences to index300If a word is repeated too many times in the same document it can slow queries down quite a bit, so truncate each term at this many occurences. If you are doing book search then you might want to make this value higher.
37mlkftmINT32linkdb min files needed to trigger to merge6Merge is triggered when this many linkdb data files are on disk. Raise this when initially growing an index in order to keep merging down.
38mtftgmINT32tagdb min files to merge2Merge is triggered when this many tagdb data files are on disk.
39mtftmINT32titledb min files needed to trigger merge6Merge is triggered when this many titledb data files are on disk.
40mpftmINT32posdb min files needed to trigger to merge6Merge is triggered when this many posdb data files are on disk. Raise this while doing massive injections and not doing much querying. Then when done injecting keep this low to make queries fast.
41mpsssINT32sortdb min files needed to trigger to merge3Merge is triggered when this many sortdb data files are on disk.
42mtftspINT32spiderdb min files needed to trigger to merge4Merge is triggered when this many spiderdb data files are on disk.
43eldbBOOL (0 or 1)enable linkdb1If this is true Gigablast will index hyper-link text and use hyper-link structures. Link voting as mentioned below requires this data in order to work. If you will never be doing link voting then you can disable this and never index the link information to save disk space.
44gltBOOL (0 or 1)enable link voting1If this is true Gigablast will use the indexed hyper-link structures in linkdb to boost the quality of indexed documents. You can disable this when doing a ton of injections to keep things fast. Then do a posdb (index) rebuild after re-enabling this when you are done injecting. Or if you simply do not want link voting this will speed up your injections and spidering a bit.
45csniBOOL (0 or 1)compute inlinks to sites1If this is true Gigablast will compute the number of site inlinks for the sites it indexes. This is a measure of the sites popularity and is used for ranking and some times spidering prioritzation. It will cache the site information in tagdb. The greater the number of inlinks, the longer the cached time, because the site is considered more stable. If this is NOT true then Gigablast will use the included file, sitelinks.txt, which stores the site inlinks of millions of the most popular sites. This is the fastest way. If you notice a lot of getting link info requests in the sockets table you may want to disable this parm.
46dlscBOOL (0 or 1)do link spam checking1If this is true, do not allow spammy inlinks to vote. This check is too aggressive for some collections, i.e. it does not allow pages with cgi in their urls to vote.
47ovpidBOOL (0 or 1)restrict link voting by ip1If this is true Gigablast will only allow one vote per the top 2 significant bytes of the IP address. Otherwise, multiple pages from the same top IP can contribute to the link text and link-based quality ratings of a particular URL. Furthermore, no votes will be accepted from IPs that have the same top 2 significant bytes as the IP of the page being indexed.
48uvfFLOAT32update link info frequency60.000000How often should Gigablast recompute the link info for a url. Also applies to getting the quality of a site or root url, which is based on the link info. In days. Can use decimals. 0 means to update the link info every time the url's content is re-indexed. If the content is not reindexed because it is unchanged then the link info will not be updated. When getting the link info or quality of the root url from an external cluster, Gigablast will tell the external cluster to recompute it if its age is this or higher.
49dsdBOOL (0 or 1)do serp detection1If this is eabled the spider will not allow any docs which are determined to be serps.
50mtdlINT32max text doc length1048576Gigablast will not download, index or store more than this many bytes of an HTML or text document. XML is NOT considered to be HTML or text, use the rule below to control the maximum length of an XML document. Use -1 for no max.
51modlINT32max other doc length1048576Gigablast will not download, index or store more than this many bytes of a non-html, non-text document. XML documents will be restricted to this length. Use -1 for no max.
52aftBOOL (0 or 1)apply filter to text pages0If this is false then the filter will not be used on html or text pages.
53confilterSTRINGfilter nameProgram to spawn to filter all HTTP replies the spider receives, taking input as first arg (max length 64). Leave blank for none.
54ftoINT32filter timeout40Kill filter shell after this many seconds. Assume it stalled permanently.
55upthBOOL (0 or 1)use pdftohtml and other filters1This cores a lot, so we have an option to turn it off. Also affects antiword, excel and other filters.
56mitBOOL (0 or 1)make image thumbnails0Try to find the best image on each page and store it as a thumbnail for presenting in the search results.
57mtwhINT32max thumbnail width or height250This is in pixels and limits the size of the thumbnail. Gigablast tries to make at least the width or the height equal to this maximum, but, unless the thumbnail is sqaure, one side will be longer than the other.
58isrBOOL (0 or 1)index spider status documents0Index a spider status "document" for every url the spider attempts to spider. Search for them using special query operators like type:status or gberrorstr:success or stats:gberrornum to get a histogram. See syntax page for more examples. They will not otherwise show up in the search results.
59ahtsubhBOOL (0 or 1)assign spider host by url hash0Delegate spidering by url hash rather than by first ip. Good for collections which are only spidering one domain.
60ibBOOL (0 or 1)index body1Index the body of the documents so you can search it. Required for searching that. You wil pretty much always want to keep this enabled. Does not apply to JSON documents.
61usubSTRINGurl find and replaceMatch all spidered urls to transform them. The format is space separated tuples: match_regex replace_val, one per line. If the replace val is omitted then it will be replaced with nothing. Back references are permitted using \1 - \9. For example:
\?.*
would remove all cgi parms.
(https?://)www\. \1mx\.
Would replace all www. sites with mx. sites.
62apiUrlSTRINGdiffbot api urlSend every spidered url to this url and index the reply in addition to the normal indexing process. Example: by specifying http://api.diffbot.com/v3/analyze?mode=high-precision&token= here you can index the structured JSON replies from diffbot for every url that is spidered. Gigablast will automatically append a &url= to this url before sending it to diffbot.
63urlProcessPatternTwoSTRINGdiffbot url process patternOnly send urls that match this simple substring pattern to Diffbot. Separate substrings with two pipe operators, ||. Leave empty for no restrictions.
64urlProcessRegExTwoSTRINGdiffbot url process regexOnly send urls that match this regular expression to Diffbot. Leave empty for no restrictions.
65pageProcessPatternTwoSTRINGdiffbot page process patternOnly send urls whose content matches this simple substring pattern to Diffbot. Separate substrings with two pipe operators, ||. Leave empty for no restrictions.



/admin/proxies - proxies   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3eliteproxyipsSTRINGelite spider proxy ipsLike spider proxies, but these are more expensive, so we don't send test queries.
4maxproxiestotryperipwhenconfidentINT32max proxies to try per ip (when confident of ban)10If we detect a proxy is banned how many more do we try before giving up. This is only when we are confident the proxy IP is banned. Like when we detect certain captchas. However, a lot of pages have captchas to prevent automated access and it does not mean your IP is banned.
5maxproxiestotryperppwhenunconfidentINT32max proxies to try per ip (when UNconfident of ban)2If we detect a proxy is banned how many more do we try before giving up. This is only when we are not so confident the proxy IP is banned, but it could be. Like, for instance, a TCP timedout or connection refused, or 403 Permission Denied response. All of which may or may not indicate a ban.
6minipcrawldelayperproxyMSINT32min ip crawl delay per proxy288000Each proxy waits at least this many milliseconds between downloads for urls from the same IP. Does not apply to super proxies (only 1 proxy of a given type specified).
7proxytesturlSTRINGspider proxy test urlhttp://www.gigablast.com/Download this url every minute through each proxy listed above to ensure they are up. Typically you should make this a URL you own so you do not aggravate another webmaster.
8proxytesttimeINT32spider proxy test ping time60Download the proxy test url once every this many seconds. Set to 0 to disable
9resetproxytableUNARY CMD (set to 1)reset proxy tableReset the proxy statistics in the table below. Makes all your proxies treated like new again.
10userandagentsBOOL (0 or 1)mix up user agents1Use random user-agents when downloading through a spider proxy listed above to protecting gb's anonymity. The User-Agent used is a function of the proxy IP/port and IP of the url being downloaded. That way it is consistent when downloading the same website through the same proxy.
11proxyAuthSTRINGsquid proxy authorized usersGigablast can also simulate a squid proxy, complete with caching. It will forward your request to the proxies you list above, if any. This list consists of space-separated username:password items. Leave this list empty to disable squid caching behaviour. The default cache size for this is 10MB per shard. Use item *:* to allow anyone access.
12alwaysuseproxysitesSTRINGsites to always use proxiesfacebook.com linkedin.com amazon.com yelp.com irs.gov quora.com stackexchange.comalways use proxies when hitting these sites. The collection must have always use proxies or automatically use proxies enabled before gb will use proxies for the listed sites.
13alwaysuseeliteproxysitesSTRINGsites to always use elite proxiesalways use elite proxies when hitting these sites. The collection must have always use proxies or automatically use proxies enabled before gb will use the elite proxies for the listed sites.
14userobotstxtexceptionsSTRINGsites to not use robots.txtSites in this list do not use robots.txt, even if that setting is enabled at the collection level.
15userelnofollowexceptionsSTRINGsites to not respect rel nofollow on linksSites in this list still follow links, even when rel=nofollow is set.
16usecanonicalexceptionsSTRINGsites to not use canonical redirectsSites in this dont redirect on rel=canonical, even if it is enabled



/admin/log - log controls   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3hrBOOL (0 or 1)log http requests1Log GET and POST requests received from the http server?
4laqBOOL (0 or 1)log autobanned queries1Should we log queries that are autobanned? They can really fill up the log.
5lqttINT32log query time threshold5000If query took this many millliseconds or longer, then log the query and the time it took to process.
6lqrBOOL (0 or 1)log query reply0Log query reply in proxy, but only for those queries above the time threshold above.
7lsuBOOL (0 or 1)log spidered urls1Log status of spidered or injected urls?
8lncBOOL (0 or 1)log network congestion0Log messages if Gigablast runs out of udp sockets?
9liBOOL (0 or 1)log informational messages1Log messages not related to an error condition, but meant more to give an idea of the state of the gigablast process. These can be useful when diagnosing problems.
10llbBOOL (0 or 1)log limit breeches0Log it when document not added due to quota breech. Log it when url is too long and it gets truncated.
11ldaBOOL (0 or 1)log debug admin messages0Log various debug messages.
12ldbBOOL (0 or 1)log debug build messages0
13ldciBOOL (0 or 1)log debug crawl info messages1
14lddBOOL (0 or 1)log debug database messages0
15lddmBOOL (0 or 1)log debug dirty messages0
16lddiBOOL (0 or 1)log debug disk messages0
17ldpcBOOL (0 or 1)log debug disk page cache0
18lddnsBOOL (0 or 1)log debug dns messages0
19ldhBOOL (0 or 1)log debug http messages0
20ldiBOOL (0 or 1)log debug image messages0
21ldlBOOL (0 or 1)log debug loop messages0
22ldgBOOL (0 or 1)log debug language detection messages0
23ldliBOOL (0 or 1)log debug link info0
24ldmBOOL (0 or 1)log debug mem messages0
25ldmuBOOL (0 or 1)log debug mem usage messages0
26ldnBOOL (0 or 1)log debug net messages0
27ldqBOOL (0 or 1)log debug query messages0
28ldqtaBOOL (0 or 1)log debug quota messages0
29ldrBOOL (0 or 1)log debug robots messages0
30ldsBOOL (0 or 1)log debug spider cache messages0
31ldspBOOL (0 or 1)log debug speller messages0
32ldsccBOOL (0 or 1)log debug sections messages0
33ldserpBOOL (0 or 1)log debug serp cache0
34ldsiBOOL (0 or 1)log debug seo insert messages0
35ldseoBOOL (0 or 1)log debug seo messages0
36ldstBOOL (0 or 1)log debug stats messages0
37ldsuBOOL (0 or 1)log debug summary messages0
38ldspidBOOL (0 or 1)log debug spider messages0
39ldspmthBOOL (0 or 1)log debug msg13 messages0
40dmthBOOL (0 or 1)disable host0 for msg13 reception hack0
41ldsprBOOL (0 or 1)log debug spider proxies0
42ldspuaBOOL (0 or 1)log debug url attempts0
43ldsdBOOL (0 or 1)log debug spider downloads0
44ldfbBOOL (0 or 1)log debug facebook0
45ldtmBOOL (0 or 1)log debug tagdb messages0
46ldtBOOL (0 or 1)log debug tcp messages0
47ldtbBOOL (0 or 1)log debug tcp buffer messages0
48ldthBOOL (0 or 1)log debug thread messages0
49ldtiBOOL (0 or 1)log debug title messages0
50ldtimBOOL (0 or 1)log debug timedb messages0
51ldtoBOOL (0 or 1)log debug topic messages0
52ldtopdBOOL (0 or 1)log debug topDoc messages0
53lduBOOL (0 or 1)log debug udp messages0
54ldunBOOL (0 or 1)log debug unicode messages0
55ldwlcBOOL (0 or 1)log debug winner list cache1
56ldreBOOL (0 or 1)log debug rebuild messages0
57ldpdBOOL (0 or 1)log debug pub date extraction messages0
58ltbBOOL (0 or 1)log timing messages for build0Log various timing related messages.
59ltadmBOOL (0 or 1)log timing messages for admin0Log various timing related messages.
60ltdBOOL (0 or 1)log timing messages for database0
61ltnBOOL (0 or 1)log timing messages for network layer0
62ltqBOOL (0 or 1)log timing messages for query0
63ltspcBOOL (0 or 1)log timing messages for spcache0Log various timing related messages.
64lttBOOL (0 or 1)log timing messages for related topics0
65lrBOOL (0 or 1)log reminder messages0Log reminders to the programmer. You do not need this.



/admin/masterpasswords - master passwords   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.



/admin/addcoll - add a new collection   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3addcollUNARY CMD (set to 1)add collectionAdd a new collection with this name. No spaces allowed or strange characters allowed. Max of 64 characters. REQUIRED



/admin/delcoll - delete a collection   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3delcollUNARY CMD (set to 1)delete collectionDelete the specified collection. You can specify multiple &delcoll= parms in a single request to delete multiple collections at once. REQUIRED



/admin/clonecoll - clone one collection's settings to another   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionClone settings INTO this collection. REQUIRED
4clonecollUNARY CMD (set to 1)clone collectionClone collection settings FROM this collection. REQUIRED



/admin/rebuild - rebuild data   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3rmeBOOL (0 or 1)rebuild mode enabled0If enabled, gigablast will rebuild the rdbs as specified by the parameters below. When a particular collection is in rebuild mode, it can not spider or merge titledb files.
4rctrSTRINGcollection to rebuildName of collection to rebuild. REQUIRED
5racBOOL (0 or 1)rebuild ALL collections0If enabled, gigablast will rebuild all collections.
6rmtuINT32memory to use for rebuild200000000In bytes.
7mrpsINT32max rebuild injections1Maximum number of outstanding injections for rebuild.
8rfrBOOL (0 or 1)full rebuild0If enabled, gigablast will reinject the content of all title recs into a secondary rdb system. That will the primary rdb system when complete.
9rfrknsxBOOL (0 or 1)add spiderdb recs of non indexed urls0If enabled, gigablast will add the spiderdb records of unindexed urls when doing the full rebuild or the spiderdb rebuild. Otherwise, only the indexed urls will get spiderdb records in spiderdb. This can be faster because Gigablast does not have to do an IP lookup on every url if its IP address is not in tagdb already.
10rrliBOOL (0 or 1)recycle link text1If enabled, gigablast will recycle the link text when rebuilding titledb. The siterank, which is determined by the number of inlinks to a site, is stored/cached in tagdb so that is a separate item. If you want to pick up new link text you will want to set this to NO and make sure to rebuild titledb, since that stores the link text. If you recycle link text the rebuild might be around twice as fast.
11rrtBOOL (0 or 1)rebuild titledb0If enabled, gigablast will rebuild this rdb
12rriBOOL (0 or 1)rebuild posdb1If enabled, gigablast will rebuild this rdb, ths index.
13rrclBOOL (0 or 1)rebuild clusterdb0If enabled, gigablast will rebuild this rdb
14rrspBOOL (0 or 1)rebuild spiderdb0If enabled, gigablast will rebuild this rdb
15rrldBOOL (0 or 1)rebuild linkdb0If enabled, gigablast will rebuild this rdb
16ruruBOOL (0 or 1)rebuild root urls1If disabled, gigablast will skip root urls.
17runruBOOL (0 or 1)rebuild non-root urls1If disabled, gigablast will skip non-root urls.



/admin/inject - inject url in the index here   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionInject into this collection. REQUIRED
4urlSTRINGurlSpecify the URL that will be immediately crawled and indexed in real time while you wait. The browser will return the final index status code. Alternatively, use the add url page to add urls individually or in bulk without having to wait for the pages to be actually indexed in realtime. By default, injected urls take precedence over the "insitelist" expression in the url filters so injected urls need not match the patterns in your site list. You can change that behavior in the url filters if you want. Injected urls will have a hopcount of 0. The injection api is described on the api page. Make up a fake url if you are injecting content that does not have one.

If the url ends in .warc or .arc or .warc.gz or .arc.gz Gigablast will index the contained documents as individual documents, using the appropriate dates and other meta information contained in the containing archive file. REQUIRED
5qtsSTRINGquery to scrapeScrape popular search engines for this query and inject their links. You are not required to supply the url parm if you supply this parm.
6injectlinksBOOL (0 or 1)inject links0Should we inject the links found in the injected content as well?
7spiderlinksBOOL (0 or 1)spider links0Add the outlinks of the injected content into spiderdb for spidering?
8newonlyBOOL (0 or 1)only inject content if new0If the specified url is already in the index then skip the injection.
9deleteurlBOOL (0 or 1)delete from index0Delete the specified url from the index.
10recycleBOOL (0 or 1)recycle content0If the url is already in the index, then do not re-download the content, just use the content that was stored in the cache from last time.
11dedupBOOL (0 or 1)dedup url0Do not index the url if there is already another url in the index with the same content.
12urlipIPurl IP0.0.0.0Use this IP when injecting the document. Do not use or set to 0.0.0.0, if unknown. If provided, it will save an IP lookup.
13hasmimeBOOL (0 or 1)content has mime0If the content of the url is provided below, does it begin with an HTTP mime header?
14delimSTRINGcontent delimeterIf the content of the url is provided below, then it consist of multiple documents separated by this delimeter. Each such item will be injected as an independent document. Some possible delimeters: ======== or <doc>. If you set hasmime above to true then Gigablast will check for a url after the delimeter and use that url as the injected url. Otherwise it will append numbers to the url you provide above.
15contenttypeSTRINGcontent typetext/htmlIf you supply content in the text box below without an HTTP mime header, then you need to enter the content type. Possible values: text/html text/plain text/xml application/json
16charsetINT32content charset106A number representing the charset of the content if provided below and no HTTP mime header is given. Defaults to utf8 which is 106. See iana_charset.h for the numeric values.
17contentSTRINGcontentIf you want to supply the URL's content rather than have Gigablast download it, then enter the content here. Enter MIME header first if "content has mime" is set to true above. Separate MIME from actual content with two returns. At least put a single space in here if you want to inject empty content, otherwise the content will be downloaded from the url. This is because the page injection form always submits the content text area even if it is empty, which should signify that the content should be downloaded.
18metadataSTRINGmetadataJson encoded metadata to be indexed with the document.
19sectionsBOOL (0 or 1)get sectiondb voting info0Return section information of injected content for the injected subdomain.
20diffbotreplySTRINGdiffbot replyUsed exclusively by diffbot. Do not use.



/admin/addurl - add url page for admin   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionAdd urls into this collection. REQUIRED
4urlsSTRINGurls to addList of urls to index. One per line or space separated. If your url does not index as you expect you can check it's spider history by doing a url: search on it. Added urls will match the isaddurl directive on the url filters page. The add url api is described on the api page. REQUIRED
5stripBOOL (0 or 1)strip sessionids1Strip added urls of their session ids.
6spiderlinksBOOL (0 or 1)harvest links1Harvest links of added urls so we can spider them?.
7stripINT32hopcount255Hopcount of the url. How far is it from the seeds, measured in hops.



/admin/reindex - query delete/reindex   [ show parms in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.
3cSTRINGcollectionquery reindex in this collection. REQUIRED
4qSTRINGquery to reindex or deleteWe either reindex or delete the search results of this query. Reindexing them will redownload them and possible update the siterank, which is based on the number of links to the site. This will add the url requests to the spider queue so ensure your spiders are enabled. REQUIRED
5srnINT32start result number0Starting with this result #. Starts at 0.
6ernINT32end result number99999999Ending with this result #. 0 is the first result #.
7qlangSTRINGquery languageenThe language the query is in. Used to rank results. Just use xx to indicate no language in particular. But you should use the same qlang value you used for doing the query if you want consistency.
8qrecycleBOOL (0 or 1)recycle content0If you check this box then Gigablast will not re-download the content, but use the content that was stored in the cache from last time. Useful for rebuilding the index to pick up new inlink text or fresher sitenuminlinks counts which influence ranking.
9forcedelBOOL (0 or 1)FORCE DELETE0Check this checkbox to delete the results, not just reindex them.
10quickdelBOOL (0 or 1)QUICK DELETE0Check this checkbox to INSTANTLY delete the results, not just reindex them. You'll have to do a tight merge on posdb and on titledb to realize the deletes. This is currently only available to crawlbot custom crawls.
11forceupdateBOOL (0 or 1)Force Update0Check this to ignore processing errors or duplicate detection to update the results unconditionally.



/admin/stats - general statistics   [ show parms in xml or json ]   [ show status in xml or json ]

Input
#ParmTypeTitleDefault ValueDescription
1 formatSTRINGoutput formathtmlDisplay output in this format. Can be html, json or xml.
2 showinputBOOL (0 or 1)show input and settings1Display possible input and the values of all settings on this page.