YaCy Release 1.3

update to pdf parser

extended the Scheduler: introduced scheduled events
- an event type (once, regular) can be selected
- for this event type, a fixed time can be selected. This may be either
directly after startup or at one of the full hours at a day (==25
options)
The main point about this feature is the opportunity to start an action
directly after startup. That makes it possible to create YaCy
distributions which, after started at the first time, start to index
parts of the intranet/internet by itself.

using the author field as solr-native facet. this makes it necessary to
introduce a copy-field for the author field to be copied to a string
field. This field is then used to generate facets. Without this field,
the facet would consist only of the words of the author names, not of
the full author string.

- added a new solr field references_i which stores the number of
INCOMING links to the corresponding web page. This information is taken
from the reverse link index (a 'little sister' of the RWI index).
- this field can be of use to enhance the ranking because a web page
with more incoming links can be more more important than others. But
this is not true for typical link pages like menues. Therefore the
number of outgoing links is needed.
- added a new solr attribute 'bf' to solr queries which is a boost
function extension. this field can contain a formula which comuptes the
boost according to given field values. After some experiments the
following forumla is now default:
div(add(1,references_i),pow(add(1,inboundlinkscount_i),1.6))^0.4
This takes the number of references and the inbound links. Further
experiments are needed to enhance that forumula.

removed dependency of vocabulary navigation from Jena and it's
triplestore; the vocabulary search is now done using generic solr fields
which are created on-the-fly during runtime.

distinguishing modified query string and original query string

added a Boost class which stores solr query boost values. The class can
be configured using the yacy.init file. The boost information is taken
from the configuration each time when a query to solr is done.

using more pre-compile pattern for split methods

- added a field cache for solr queries which call only for a single
value
- fixed a version conflict exception within a solr add request

- added another enumeration method in kelondro data structure to get a
more random access to data for the balancer
- added random access inside the balancer

removed overhead by preventing generation of full search results when
only the url is requested

added a feature to find similarities in documents.
This uses an enhanced version of the Nutch/Solr TextProfileSignatue.
As a result, a signature of the document is written to the solr search
index. Additionally for each time when a signature is written, it is
checked if the singature exists already in the index. If the signature
does not exist, the document is marked as unique. The unique attribute
can now be used to sort document lists and bring duplicates to the end
of a result list.
To enable this, a large portion of the search api to Solr had to be
changed. This affected mainly caching of 'exists' searches to enhance
the check for existing signatures and do this without actually doing a
solr query.
Because here the first time a long number is used as value in the Solr
store, also the value naming in the YaCySchema had to be adopted and
normalized. This caused that many files had to be changed.

- added field options to all solr queries. This can be used to restrict
the actual data which is fetched from solr.
- used the new field options to reduce generic options like getting the
load date or the count of search results. should increase overall speed
- used the new field options to reduce overhead in the host browser
during aquisition of links.
- used the field options to make checking of links in crawler faster
- if the crawler is paused, the crawl queue is not cleaned

Merge commit '2bb8f045cc92f31fc7e720cc30b38af417563890'

redesign of the QueryParams class: introduced QueryGoal which holds the
query string parser. This shall be used to create a proper full-string
matching which is handled then by QueryGoal.

added deletion of hosts during crawl start if deleteold option was given

because we have the inurl:<term> - searchmodifier, we don't actually
need regular expressions as search attributes. They had now been removed
from the advanced search page while they are still created internally.
The filter is then expressed against solr as regular expression filter
query. If the expression points out a selection of an specific protocol,
host or filetype this is then translated into a facetted query.

update to search tests (use yacy interface and a bugfix)

fix for smb crawl situation (lost too many urls)

fix SeedUpload setting propery name for include template file

- apply fix for localhost handling (from yacy2solr) also to metadata2solr 

fix: exception if default work files don't exist

fix for event starter: delete start time when event is removed

fix for config basic: do not accept empty peer names

fix: set defaul language to "en"

quickfix for translated link containig word "browse" in ru & uk, see http://bugs.yacy.net/view.php?id=213

fix for bad xml in gsa result when doing a query with quotes

fix for wrong display of error urls in HostBrowser

fix for waitingtime computation for intranet configuration

patch for funny symbols in url paths (like tilde)

fix: prevent regex pattern compile error for blacklist import for path '*' (extend it to '.*')

fix: respect config setting of "show Nav Top-Menu" in HostBrowser.html for public users (as hostbrowser is now available in search results)

added debug code to crawler monitor

fixes for filesystem indexing

fix: remove fixed individual testing IP (85.25.151.30 = server4you.de) from default/yacy.network.freeworld.unit

Release 1.3

updated slf4j and log4j

use the search configuration to default the cacheStrategy to the value
as given in the search configuration

use solr boost configuration to select search fields. At this time it is
possible to enter a negative boost value to switch that value off. This
might be different in the future with a better input interface.

- made image search in interactive search using the ViewImage servlet -
that enables viewing of images for intranet SMB servers.
- added a filter search for protocol, tld and ext again; otherwise p2p
search produces a lot of rubbish

SeedUpload url : check to reject localhost url included in saveSeedList (same check as in / copied from Seed.isProper() ), to prevent identity change on next startup (due to rejected seeduploadurl).

copy work tables from defaults/data/work if exist there and not in
DATA/WORK
This can be used to create start-up behavior work scripts in the
api.bheap table

removed protocol, tld, ext from the urlmask and created specific
navigation field for these

search process enhancements

- removed all extension types from extension navigation which are not
proper/known
- automatically show the protocol navigation if there is more than http
and https
- automatically show the extension navigation if there is some media
content

using the publisher information for the author field if no author is
given. This applies to cases where only the copyright field in the html
header is filled but not the author field

- using a filter query for facet restriction
- calculating the whole search result in at most two sub-queries from
solr

using the solr facets as navigation in yacyinteractive.html instead of
counting locally result types

added another solr field clickdepth_i which reflects the number of
clicks which are necessary to get from the portal of a host to a
specific document. At this time, only the start document is flagged with
clickdepth '0', all other with '-1'. To get the actual clickdepth, a
process must use crawled information to collect the actual number of
clicks. This will be added in another/next step.

- fix for localhost detection
- added IPv6 patterns for localhost detection

PerformanceQueues: disable input for hardcoded httpd performance values

- fixes for host navigation
- fixes for filetype navigation
- removed unused code

- fixed 'delete from subpath' during crawl start which deleted nothing;
now works;
- changed some crawl start html design details

fixes in the Russian translation, chmod a-x cn.lng

if maxFileSize < 0 then the file size limit is without limit.

more search command tools

you can now search for '*' to get just ALL entries in the search index
as result list. This makes sense if you intend to search just by using
the navigation tools to cut the data set into navigation 'slices'.

allow larger no-proxy expressions

you can now search for '*' to get just ALL entries in the search index
as result list. This makes sense if you intend to search just by using
the navigation tools to cut the data set into navigation 'slices'.

re-integrating useForHost method (lost sometime?) to get the noProxy
pattern working again. Without using this method all remote urls
including the localhost had been accessed through the configured proxy

fix Servlet template on conditional file include with use of conditional template pattern in included template file (example IndexCreateQueues_p.html)
see bug http://bugs.yacy.net/view.php?id=215

- fix for bad url conversion in bookmarks when using smb urls
- fix for localhost hosts in solr schema host handling

- making blacklist path part case insensitive (solving http://bugs.yacy.net/view.php?id=171)
- blacklist test adding explicite response text "not blocked" if no blacklist match

remove NOT NEEDED reference to solr.YaCySchema from ConfigurationSet to be able to use ConfigurationSet for other conf files (than solr.keys.default.list).

introduced a better place to update the lastacc time value in latency

removed Latency update after URL selection because that causes
a completely wrong behaviour when cache fresh cases appear. Makes
re-crawling MUCH faster!

- clear the search cache when altering the solr boosts
- better positions for submit buttons

using a filter query for the site parameter in GSA api

latency fix: only set last-visit time if access was actually by the
robot

added another blacklist-cleaner into balancer

- check blacklist (again) when taking urls from the crawl stack because
the blacklist may get extended during crawling
- removed debug output

more robustness during shutdown

Brute-force attempt to start solr in case of a memory problem.
I don't actually know if this is correct. It is a desperate try to get
YaCy running on production servers which must get alive even with
strange hacks like this. This is also related to a forum posting in
http://forum.yacy-websuche.de/viewtopic.php?t=4528&p=27135#p27135

update to Solr Boost handling

Added a new servlet to configure the solr ranking using field boosts

renamed Ranking_p.html to RankingRWI_p.html
because there will be another Ranking servlet as well at next

enhanced exists()-method for solr; should reduce a lot of IO during DHT
target selection

added number of characters in url to default index to be able to use
this field for ranking

added more logging to get info which url causes performance problems

prevent Solr "version conflict" on update by set Solr "_version_" field to 0 (=no version check)

improvements in GSA result writer

replaced more split and replaceAll missing pattern pre-compilation with
pre-compiled pattern

enhanced search result processing behavior
- query less at one time; query more often
- in between the small queries, evaluate results
- remove fields from search results which are not needed

fix: display and calculate authors and namespace search navigator if configured (otherwise skip overhead)
(leave hosts, topics and  not in ConfigPortal included filetype,  protocoll navigator untouched) 

added link to
http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html
to the /RegexTest.html servlet

start the local search only if this peer is doing a remote search or
when it is doing a local search and the peer is old

- removed multi-add of documents (no used)
- inserted specialized code for size request

introduced more structure in HostBrowser, table view, better counting,
distinguishing of error cases (fail/excluded)

added a new fail type attribute for the index to distinguish two
separate fail types: network fail and forced exclusion (i.e. by robots
or forwarding rules).

- using edismax in gsa interface
- generating less field data for gsa search results
- using a boost query in gsa interface to move double content to the end
of the result list

Merge remote-tracking branch 'reger/master'

Merge remote-tracking branch 'regerdev/master'

FINALLY YaCy can now search for full strings using double- or
singlequoted strings in the search query line!!!

content control: use up-to-date definitions

- redesign of crawl start servlet
- for domain-limited crawls, the domain is deleted now by default before
the crawl is started

- removed scheduled crawling options in crawl start because it is
superfluous there; it can be changed in the scheduler servlet. It's also
confusing in the presence of the delete-option, which will be
implemented next.
- removed unused crawl start servlet
- some refactoring to make the time parser reusable

SMW Import: replaced JSON import routines with stable ones

removed hightlighting of search results within collections in GSA
interface

added icons and a selection for hosts with urls pending for crawler or
with errors

refactor package

remove old SMW importer which was part of the ymarks package

update and generalization of the SMW import and content control routines