YaCy Release 1.82

refactor opensearch heuristic
introduce FederateSearchManager handling search heuristic to external systems via specific FederateSearchConnectors,
which provide the query() functionallity, the translation to YaCy schema .toYaCySchema() and the search() routine to deliver results to searchevents, which is generally implemented in Abstract connector.
The manager enforces now a min 15s delay between calls to external systems.
Besides the OpensearchConnector a SolrFederateSearchConnector is available. It uses a additional config file for fieldname translation.

default heuristicopensearch.conf: 
- openbdb.com removed - seems not longer to deliver results
- config via solrconnector to  datacite.org added (large technical library archive)

bump to Solr-/Lucene-4.10.3

Added a ?don't store remote search results? option
This is intended for peers who want to participate in the P2P network but don't wish to load/fill-up their index with metadata of every received search result. 
The DHT transfer is not effected by this option (and will work as usual, so that a peer disabling the new store to index switch still receives and holds the metadata according to DHT rules).
Downside for the local peer is that search speed will not improve if search terms are only avail. remote or by quick hits in local index.

To be able to improve the local index a Click-Servlet option was added additionally.
If switched on, all search result links point to this servlet, which forwards the users browser (by html header) to the desired page and feeds the page to the fulltext-index.
The servlet accepts a parameter defining the action to perform (see defaults/web.xml, index, crawl, crawllinks)

The option check-boxes are placed in ConfigPortal.html

added experimental pdf splitting which enables YaCy to split pdfs during
parsing into individual pages and add them all using different URLs.
These constructed urls are generated from the source url with an
appended page=<pagenumber> attribute to the url get/post properties.
This will distinguish the different page entries. The search result list
will then replace the post parameter with a url anchor # mark which
causes that the original url is presented in the search result. These
URLs can be opened directly on the correct page using pdf.js which is
now built-in into firefox. That means: if you find a search hit on page
5 and click on the search result, firefox will open the pdf viewer and
shows page 5.

update to Jetty 9.2.6

update to PDFBox 1.8.8

Added a transaction interface to the snapshots: all documents in the
snapshots can now be processed with transactions using commit and
rollback commands. Furthermore, a large number of monitoring methods had
been added to check the success of transactions.

The transactions for snapshots have two main components: a rss search
API to get information about latest/oldest entries and a commit/rollback
API to move entries away from the rss results. This is done by usage of
two storage locations for the snapshots, INVENTORY and ARCHIVE. New
snapshots are placed to INVENTORY, commited snapshots move to ARCHIVE,
rollback snapshots move to INVENTORY again.

Normal Workflow:
Beside all these options below, usually it is sufficient to process data
like this:
- call
http://localhost:8090/api/snapshot.rss?state=INVENTORY&order=LATESTFIRST
- process the rss result and use the <guid> value as <urlhash> (see next
command)
- for each processed result call
http://localhost:8090/api/snapshot.json?command=commit&urlhash=<urlhash>
- then you can call the rss feed again and the commited urls are omited
from the next set of items.

These are the commands to control this:
The rss feed:
http://localhost:8090/api/snapshot.rss?state=INVENTORY&order=LATESTFIRST
http://localhost:8090/api/snapshot.rss?state=INVENTORY&order=OLDESTFIRST
http://localhost:8090/api/snapshot.rss?state=INVENTORY&order=ANY
http://localhost:8090/api/snapshot.rss?state=ARCHIVE&order=LATESTFIRST
http://localhost:8090/api/snapshot.rss?state=ARCHIVE&order=OLDESTFIRST
http://localhost:8090/api/snapshot.rss?state=ARCHIVE&order=LATESTFIRST

The feed will return a <urlhash> in the <guid> - field of the rss. This
must be used for commit/rollback:

Commit/Rollback:
http://localhost:8090/api/snapshot.json?command=commit&urlhash=<urlhash>
http://localhost:8090/api/snapshot.json?command=rollback&urlhash=<urlhash>
The json will return a property list containing the property "result"
with possible values "success" or "fail", according of the result. If an
"fail" occurs, please look into the log for further info.

Monitoring:
http://localhost:8090/api/snapshot.json?command=status
This shows the total number of entries in the INVENTORY and the ARCHIVE 
http://localhost:8090/api/snapshot.json?command=list
This will result a list of all hosts which have snapshots and the number
of entries for the hosts. Counts for INVENTORY and ARCHIVE are listed in
the porperties for "count.INVENTORY" and "count.ARCHIVE"
http://localhost:8090/api/snapshot.json?command=list&depth=2
The list can be restricted to such which have a specific depth. The list
contains then the same host names, but the count values change because
only documents at that specific crawl depth are listed
http://localhost:8090/api/snapshot.json?command=list&host=yacy.net.80
This lists all urlhashes for the given host, not only an accumulated
list of the number of entries
http://localhost:8090/api/snapshot.json?command=list&host=yacy.net.80&depth=0
This restricts the list of urlhashes for that host for the given depth
http://localhost:8090/api/snapshot.json?command=list&state=INVENTORY
http://localhost:8090/api/snapshot.json?command=list&state=ARCHIVE
This selects either the INVENTORY or ARCHIVE for all list commands,
default is ALL which means that from both snapshot directories the host
information is collected and combined. You can use the state option for
all the commands as listed above

Detailed Information:
http://localhost:8090/api/snapshot.json?command=metadata&urlhash=upiFJ7Fh1hyQ
This collects metadata information for the given urlhash. This can also
be restricted with state=INVENTORY and state=ARCHIVE to test if the
document is either in one of these snapshot directories. If an urlhash
is not found, an empty result is returned. If an entry was found and the
state was not restricted, then the result contains a state property
containing the name of the location where the document is, either
INVENTORY or ARCHIVE.

Hint:
If a very large number of documents is inside of INVENTORY, then it
could be better to call the rss feed with
http://localhost:8090/api/snapshot.rss?state=INVENTORY&order=ANY
because that is very efficient.

update to metadata-extractor-2.7.0.jar
add 2 simple JUnit test cases for jpeg and tif parsing

Added and integrated new date detection class which can identify date
notions within the fulltext of a document. This class attempts to
identify also dates given abbreviated or with missing year or described
with names for special days, like 'Halloween'. In case that a date has
no year given, the current year and following years are considered.

This process is therefore able to identify a large set of dates to a
document, either because there are several dates given in the document
or the date is ambiguous. Four new Solr fields are used to store the
parsing result:

dates_in_content_sxt:
if date expressions can be found in the content, these dates are listed
here in order of the appearances

dates_in_content_count_i:
the number of entries in dates_in_content_sxt

date_in_content_min_dt:
if dates_in_content_sxt is filled, this contains the oldest date from
the list of available dates

#date_in_content_max_dt:
if dates_in_content_sxt is filled, this contains the youngest date from
the list of available dates, that may also be possibly in the future

These fields are deactiviated by default because the evaluation of
regular expressions to detect the date is yet too CPU intensive. Maybe
future enhancements will cause that this is switched on by default.

The purpose of these fields is the creation of calendar-like search
facets, to be implemented next.

enhanced the snapshot functionality:

- snapshots can now also be xml files which are extracted from the solr
index and stored as individual xml files in the snapshot directory along
the pdf and jpg images
- a transaction layer was placed above of the snapshot directory to
distinguish snapshots into 'inventory' and 'archive'. This may be used
to do transactions of index fragments using archived solr search results
between peers. This is currently unfinished, we need a protocol to move
snapshots from inventory to archive
- the SNAPSHOT directory was renamed to snapshot and contains now two
snapshot subdirectories: inventory and archive
- snapshots may now be generated by everyone, not only such peers
running on a server with xkhtml2pdf installed. The expert crawl starts
provides the option for snapshots to everyone. PDF snapshots are now
optional and the option is only shown if xkhtml2pdf is installed.
- the snapshot api now provides the request for historised xml files,
i.e. call:
http://localhost:8090/api/snapshot.xml?urlhash=Q3dQopFh1hyQ
The result of such xml files is identical with solr search results with
only one hit.
The pdf generation has been moved from the http loading process to the
solr document storage process. This may slow down the process a lot and
a different version of the process may be needed. 

ViewFile servlet: update index if newer,
so viewed text and metadata (stored) info is similar
- to archive it, use request with profile to allow indexing (defaultglobaltext) and update index 
   (the resource is loaded, parsed anyway, so it's not a expensive operation)

Request: remove 2 unused init parameter 
- number of anchors of the parent
- forkfactor sum of anchors of all ancestors

added a servlet which can create preview images, preview tumbnails and
preview pdfs from web pages, i.e.:
http://localhost:8090/api/snapshot.png?url=http://yacy.net/en/&width=128&height=128
http://localhost:8090/api/snapshot.jpg?url=http://yacy.net/en/&width=128&height=128
http://localhost:8090/api/snapshot.pdf?url=http://yacy.net/en/

This supports also an on-the-fly generation of the preview documents if
the user is an administrator. Otherwise, the servlet fails.
To enable this, you must add wkhtmltopdf, imagemagick and (on headless
servers) xvfb to your operation system.

for detailed instructions, see
https://gitorious.org/yacy/rc1/commit/97f6089a41a4ed40aef84f692690e30f50585f5d

Replaced all fixed thread pools with cached thread pools. The cached
thread pools will flush their cached (dead) threads after 60 seconds.
This will cause that YaCy now runs constantly withl about 50 threads,
about 100 at peak times. Previously, about 400 threads had been cached
and kept in a hibernation state, which caused that the numproc counter
in /proc/user_beancounters (exists only in VM-hosted linux) was as high
as the cached number of threads. This caused that VM supervisors
terminated whole VM sessions if a limit was reached. Many VM providers
have limits of numproc=96 which made it virtually impossible to run YaCy
on such machines. With this change, it will be possible to run many YaCy
instances even on VM hosts. 

YaCy can now create web page snapshots as pdf documents which can later
be transcoded into jpg for image previews. To create such pdfs you must
do:

Add wkhtmltopdf and imagemagick to your OS, which you can do:
On a Mac download wkhtmltox-0.12.1_osx-cocoa-x86-64.pkg from
http://wkhtmltopdf.org/downloads.html and downloadh
ttp://cactuslab.com/imagemagick/assets/ImageMagick-6.8.9-9.pkg.zip
In Debian do "apt-get install wkhtmltopdf imagemagick"

Then check in /Settings_p.html?page=ProxyAccess: "Transparent Proxy" and
"Always Fresh" - this is used by wkhtmltopdf to fetch web pages using
the YaCy proxy. Using "Always Fresh" it is possible to get all pages
from the proxy cache.

Finally, you will see a new option when starting an expert web crawl.
You can set a maximum depth for crawling which should cause a pdf
generation. The resulting pdfs are then available in
DATA/HTCACHE/SNAPSHOTS/<host>.<port>/<depth>/<shard>/<urlhash>.<date>.pdf

added new web page snapshot infrastructure which will lead to the
ability to have web page previews in the search results.
(This is a stub, no function available with this yet...)

update to Jetty 9.2.4

added option to make the YaCy proxy act as the cache is never stale. If
set to 'Always Fresh' the cache is always used if the entry in the cache
exist. This is a good way to archive web content and access it without
going online again in case the documents exist.
To do so, open /Settings_p.html?page=ProxyAccess and check the "Always
Fresh" checkbox.
This is set do false which behave as set before.
If you set this to true, then you have your web archive in DATA/HTCACHE.
Copy this to carry around your private copy of the internet!

added loading of the synonyms file from addon/synonyms into the
knowledge loader

added new 'firstSeen' database table and necessary data structures which
hold a date for each URL to record when a url was first seen. This is
then used to overwrite the modification date for urls upon recrawl in
case that the first-seen date is before the latest document date. This
behaviour is necessary due to the common behaviour of content management
systems which attach always the current date to all documents. Using the
firstSeen database it is possible to approximate a real first document
creation date in case that the crawler starts frequently for the same
domain. As a result the search results ordered by date have a much
better quality and the usage of YaCy as search agent for latest news has
a better quality.

update to httpclient-4.3.6

update to solr-/lucene-4.10.2

fix for exact_signature_unique_b, exact_signature_copycount_i,
fuzzy_signature_unique_b and fuzzy_signature_copycount_i: apply same
criteria for 'valid document' as for title and description uniqueness
test.

added partial updates to solr during postprocessing: during
postprocessing the solr documents are now not completely retrieved.
instead, only fiels, needed for the postprocessing are extracted. When
Solr document are written, this is done using partial updates.

This increases postprocessing speed by about 50% for embedded Solr
configurations. For external Solr configurations the enhancement should
be much higher because the postprocessing with remote Solr is very slow.
When doing partial updates to a remote Solr, this method should perform
much better than before, it is expected that this is even much higher
than the increase with local Solr.

Added an experimental audio feedback system.
This is the first element of a new 'decoration' component which may hold
switches for different external appearance parameters.
The first switch in that context is decoration.audio (as usual in
yacy.init). This value is set to false by default, that means the audio
feedback element is switched off by default. To switch it on, set
decoration.audio = true (using /ConfigProperties_p.html). You will then
hear sounds for the following events:
- remote searches
- incoming dht transmissions
- new documents from the crawler
Sound clips are stored in htroot/env/soundclips/ which is done so
because a future implementation will read these files using the http
client and with configurable urls which will make it very easy for the
user to replace the given sounds with own sounds.

Finished implementation of UPNP:

*) will try other ports if YaCy standard ports are not available
*) distinguish between internal and external port (not sure if this
works 100%)

Still to add: propery in config to enter own external port (in case of
manually configured NAT)

IPv6-enhanced Network monitoring page

large IPv6 redesign of peer ping methods!
removed preferred IPv4 in start options and added a new field IP6 in
peer seeds which will contain one or more IPv6 addresses. Now every peer
has one or more IP addresses assigned, even several IPv6 addresses are
possible. The peer-ping process must check all given and possible IP
addresses for a backping and return the one IP which was successful when
pinging the peer. The ping-ing peer must be able to recognize which of
the given IPs are available for outside access of the peer and store
this accordingly. If only one IPv6 address is available and no IPv4,
then the IPv6 is stored in the old IP field of the seed DNA.
Many methods in Seed.java are now marked as @deprecated because they had
been used for a single IP only. There is still a large construction site
left in YaCy now where all these deprecated methods must be replaced
with new method calls. The 'extra'-IPs, used by cluster assignment had
been removed since that can be replaced with IPv6 usage in p2p clusters.
All clusters must now use IPv6 if they want an intranet-routing.

update to PDFBox 1.8.7

update to Jetty 9.2.3

changed the concurrent enumeration of query results in such a way that
it is now possible to get the results in two steps:
- first retrieve all IDs as given for a query
- then retieve each document individually

This was necessary for very large result sets where a query may run for
hours and is possibly terminated by a solr-internal timeout. This occurs
regulary during postprocessing and therefore this commit may fix
unwanted postprocessing terminations.

added a 'stats' table which records some peer statistics twice every
hour. The table can be shown with
http://localhost:8090/Tables_p.html?table=stats

The entries have the following meaning: 
aM: activeLastMonth
aW: activeLastWeek
aD: activeLastDay
aH: activeLastHour
cC: countConnected (Active Senior)
cD: countDisconnected (Passive Senior)
cP: countPotential (Junior)
cR: count of the RWI entries
cI: size of the index (number of documents)

The entry keys are abbreviated to reduce the space in the table as the
name is written again for every row.

This is the beginning of a 'yacystats' micro-alternative als built-in
function in YaCy. Graphics may follow after some time if enough test
data is available.

fixed font size and print page generation in pdf snapshots

fix for mediawiki import

remove debug limit from commit before

fix for division by zero (rare cases)

fix "null" title in response writer for documents with multivalued title

NPE fix

fix for pdf sub-page result preparation

removed debug code

fix to wkhtmltopdf usage

fixes to wkhtmltopdf call

prevent NPE during initialization of very large vocabularies

removed debug lines

fix for division by zero

fix for image parser (there is a class missing!)

fix for a count issue in snapshot api

fixes on wkhtmltopdf

fix for vocabulary import (double term detection)

fix for Is Facet checkbox

prevent NPE on host link for to short HeuristicCfg.OpenSearchURL

fix startup stop on missing HTCACHE/SNAPSHOT directory

npe fix

fix (enable) error msg on empty query

show vocabularies in search result (in case of debugging)

security bugfix

toString fix

fix field counter for multi-fields in html writer for the solr servlet

fix for wildcard patch in search queries

fix for catchall handling in search

fix for bad table iteration

html fix

added long variables to debug output in index browser

fix of bad query generation for search facets

fix for bad query generation in doublecheck in postprocessing

emergency bugfix for 100% CPU in image drawing

fix for bad timing computation in postprocessing

more fixes and enhancements to postprocessing

enhanced debug code in host browser

fix for npe (in rare cases)

fix for literal computation

fix for broken protocol navigation

more IPv6 fixes

IPv6 fix

fix for ConnectionInfo.cleanup of server-connections

fix for bad json

npe fix

fix for api icon in yacysearch_location.html

fixed location search

more ipv6 fixes

fixed appearance of RSS icon on search result page

misc bugfixes (concurrency, memory protection)

more ipv6 bugfixes

fix for local search

more ipv6 bugfixes

fix for remote search process

fix for bad node flag setting with IPv6

ipv6 fixes for Network.html front page

more ipv6 fixes

more ipv6 fixes

fix for latest UPnP update

more IPv6 bugfixes

toString fixes

fix char encoding parameter in UrlProxy

unresolved pattern fix

ipv6 fixes

fix for xss bugs found by CTF365

IPv6 host parsing bugfixes

update German translation http://mantis.tokeek.de/view.php?id=447

fix for crawl limit for number of pages fail

fix for Mac.app config - but does still not run. looking for build bug.

Release 1.82

reverted 'do not show all options' strategy. This is actually confusing
new users. Will be activated maybe again if there is an optional
tutorial mode which can be switched on for this special purpose of
running a tutorial.

a test with http://validator.w3.org/feed/#validate_by_input shows that
the time format was wrong; we must use RFC-822

Add option for extended search (Autosearch) to Bookmark.html asking all connected peers for the searchterm added as description to the bookmark created by the bookmark icon.
Intended for searches/research projects with not sufficient results from local and DHT selected remote target peers.

Function: the process checks newly created bookmarks for description starting with "query=..." and takes this to ask every peer for 20 search results and adds it to the local index in a background job.
link to start/stop the process added to /Bookmarks.html

Add title import for bookmark icon
if avail in index

- add javadoc to busythread with hint about the init parameter useage
- remove obsolete 10_httpd config parameter

documents pushed over the api/push_p.html interface will have their
unique flag set by default

better scale

do not write frame links to webgraph

revert clickservlet 
(default was indeed a mistakenly)

do not use the clickservlet by default. From my personal view, this
technique should not be used at all! This project is about privacy, the
existence of a click servlet is one example why people should NOT use a
search portal if such exists. 

please commit new files under your own name, this file was not created
by me.

added url to bookmark icon link
url is anyway needed, saves index lookup and works w/o commited url.
Removed unused order parameter

fix NPE in viewimage

Caused by: java.lang.NullPointerException
	at net.yacy.peers.graphics.EncodedImage.<init>(EncodedImage.java:73)
	at ViewImage.respond(ViewImage.java:156)

fix ConfigPortal jumps to iframe focus
add focus parameter to yacysearch.html too

add info text to metadata page (htmlresponsewriter) on no documents found

improve TexParser.mimeOf( fileextension ) by returning 1st defined in supported list.
This prevents unusual mapping of supported fileextension -> mimetype
(like htm=application/x-tex)

do not write iframe and embed links into webgraph, but use them anyway
for crawling

Fix for progress table background not resizing
when the post-processing started/ended.

adjustments for Bookmark icon to act on BookmarkDB,
it acts on YMarks but YMark interface seems not maintained,
for future features (e.g. query memory) BookmarkDB is the likely choice to expand, besides the crawlstart bookmark also the result bookmark icon now adds to BookmarkDB.
The YMark related code is (for now) left untouched so both tables are updated.

remove obsolete config footer option (ConfigPortal user.login)
no footer or footer-option in use

remove unused yacy.init item allowUnlimitedReceiveIndexFrom

reacivated clear stacks code for termination of all crawls because this
did not work wihtout that part of the code

do not flush non-errors to stdout because this is a concurrency issue.
the flush-call appeared very often in thread dumps with high load, so
this hopefully gives some performances

do not translate gif images into png images for thumbnails. Instead,
stream the original to the search result thumb viewer. This has two
reasons:
- animated gifs cause 100% cpu and deadlocks in the jvm gif parser; a
known bug which is obviously not yet fixed
- animated gifs now appear in the search result also as animation

automatically set the Q flag for smb/ftp start urls (split pdf support)

automatically swith on query option in case intranet protocols (smb/ftp)
are used. This supports the new split-pdf option.

Applied URL-decoding prior to HTML-encoding.
This removes percent-encoding from text shown in HTML

Postprocessing progress bar fix
(Make it work as [probably] actually intended)

Init Jetty using setDefaultDescriptor (web.xml) to defaults/web.xml
so web.xml in defaults dir is applied first and optional DATA/SETTINGS/web.xml loaded on top.
By using this Jetty feature (default web.xml) we assure that changes to the default are applied to existing installations
and individual addition/changes are still respected.

adjust fieldtype and description of field httpstatus_redirect_s in CollectionSchema
- the field is not used (delete candidate)

fix NPE related 500 (Bad Request) response of UrlProxy on blacklisted urls,
by adding parameter HTTPDeamon and removing unused hostAddress lookup code in sendRespondError

improve yacysearchitem, 
prevent allocation of String (modifyURL) if feature not used

add xmpcore as direct dependency to pom
(otherwise it's looked up at pdfbox archive path and not found there)

crawling of multi-page pdfs with artificial post part on smb or ftp
shares is not possible with the disabled setting; this is not temporary
disabled until a better solution is on the hand.

fix div by 0 in hello
Caused by: java.lang.ArithmeticException: / by zero
	at hello.respond(hello.java:159)

update to SLF4J 1.7.9

fix proxy redirect (http status 302) response
fixes http://mantis.tokeek.de/view.php?id=517

The url given in bug report uses a gzip input stream which causes the HTTPClient.writeto() throw an IOException due to incomplete input stream. This in turn prevents the 302 reponse to the client browser. 
By limiting to serve target content just on httpstatus=200 will proxy the header reponse and client browsers redirect settings can be honored.

enhanced initialization of autotagging

add hint to Heuristics Config on "Greedy Learning Mode" in portal config,
to point to a option to make this setting permanent.

update to commons-fileupload-1.3.1.jar 
(includes a security fix)

changed prefer strategy for http unique in such a way that http is
preferred over https. While this is a bad idea from the standpoint of
security it is more common applicable for environments where http and
https mix and for some domains https is not available. Then the
double-check is possible even if no postprocessing is performed.

fix to prevent assertion error in ranking servlet if no vocabularies are
present that could be evaluated

the miss cache does not seem to work, it sometimes contains urlhashes
from documents which actually are inside the index. This can be
reproduced using the crawl result table at 
http://localhost:8090/CrawlResults.html?process=5
The cache is temporary disabled to remove the bad behaviour, however a
later reactivation of that feater may be possible. 

fix refactored size() -> filesize() in YMarkMetadata

refactor size() -> filesize() of URIMetadataNode
(harmonize with ResultEntry and to not get confused with Collection.size())

remove redundant caching of urlhash in URIMetadataNode
(is already cached in underlaying DigestURL .url)

upd pom keyword for maven-antrun-plugin

use peeraddress for link in remote crawl list
to make link work without enabled proxy

upd pom for Jetty (missing in last commit)

preventing the use of no-cache and expires in case that images are
generated dynamically which will stay static in the future. This applies
mainly to the search result favicon in front of search hits. These icons
will now be generated once, but then caches in the browser. There is
also a YaCy-internal cache for these icons which had prevented the
re-generation of the icons in YaCy, but this cache is now superfluous
since the browser should not call the servlet ViewImage again.

fixes for searches when initialization of large autotagging libraries
have not been finished

fixes to usage of no-cache: use and recognize also the no-store
directive

reduction of http requests to YaCy using the correct cache-control,
expires and last-modified headers in http response.

fix missing AppPath
upd Maven plugin versionid

include xmpcore.jar in classpath
used by metadata-extractor

Update russian translation

added query modifier 'on'. This makes it possible to search for date
occurrences within the (web) page documents (not the document
last-modified!). This works only if the solr field dates_in_content_sxt
is enabled. A search request may then have the form "term on:<date>",
like
gift on:24.12.2014
gift on:2014/12/24
* on:2014/12/31
For the date format you may use any kind of human-readable date
representation(!yes!) - the on:<date> parser tries to identify language
and also knows event names, like:
bunny on:eastern
.. as long as the date term has no spaces inside (use a dot). Further
enhancement will be made to accept also strings encapsulated with
quotes.

added (very experimental) Solr response writer for snapshot image
results

added url, date, time and page number on pdf snapshot footer

reactivated on-demand snapshot loading

add final SolrQueryRequest.close to SolrServlet

added a note that the servlet is linked using web.xml

- fix path to default heuristic.cfg
- deprecate unused ProxyServlet

add chardet.jar to Maven dependencies

fix yacy.init comment
http://mantis.tokeek.de/view.php?id=513

add the actual DateDetection class... (missed in latest commit)

enable sku as anchor in html response writer

enhanced tagging preparation speed which reduces initialization time for
very large vocabularies

refactoring date -> lastModified

added concurrent generation of snapshot pdfs

added charset detection to vocabulary reader

added character set detection library from
http://www-archive.mozilla.org/projects/intl/chardet.html

added new options to vocabulary editor:
- new switch 'isFacet' which causes that the usage of the vocabulary for
search facets is enabled or disabled. This shall be used for large
vocabularies sind searched in solr are extremely slow if facets for a
large set of alternative terms are generated
- new option to disable auto-enrichment from synonyms
- new option to add synonyms from another column when importing from csv
- automatically recognize double-occurrences in synonyms and bundling
terms for such synonyms

remove redundant null check in ResponseHeader.lastModified
added a JUnit testcase for ResponseHeader dates (using age()),
adjusted age() to pass all tests

added confirmation dialogs for row deletion

more robustness for broken table data in Table_API_p.html -- see bug
report http://mantis.tokeek.de/view.php?id=495

enhancement for clearing the crawl queue 

modified FieldReIndex to reindex queries with low number of documents first
by using a internally a score map with number of documents as score
and working through the list from low to high.

update to commons-logging-1.2

Merge origin/master

coding fixes suggested in 
http://mantis.tokeek.de/view.php?id=509
http://mantis.tokeek.de/view.php?id=510

added rss feed output to snapshot servlet which can be used to get a
list of latest/oldest entries in the snapshot database. This is an
example:
http://localhost:8090/api/snapshot.rss?depth=2&order=LATESTFIRST&host=yacy.net&maxcount=100

The properties depth, order, host and maxcount can be omited. The
meaning of the fields are:
host: select only urls from this host or all, if not given
depth: select only urls at that crawl depth or all, if not given
maxcount: select at most the given number of urls or 10, if not given
order: either LATESTFIRST to select the youngest entries, OLDESTFIRST to
select the first entries or ANY to select any

The rss feed needs administration rights to work, a call to this servlet
with rss extension must attach login credentials.

added toString() methods to feed classes which makes it possible to
export full rss feed files out of the RSSFeed class

remove the unused Request variable
(fix of  prev. commit)

Merge origin/master

added Image Events as another option to generate images with a mac if no
Ghostscript is available or does not work...

added another path for the convert command because on older Macs
ImageMagick has a different installation location

skip creation of unused Bluelist contenttransformer

showing list of all thread in threaddump using the ThreadMXBean counter
(this obviously show more threads than before?)

set Busy- and Blocking-Threads to daemon mode (they will now not prevent
YaCy from termination if still running)

show number of threads on status page

in case that loading from the cache fails, load from wkhtmltopdf without
cache using the user agent string given in the crawl profile

recognize more html file types for snapshots

get cloned crawl start parameter for snapshots

recognize more html file extensions

fix to xvfb-run usage (quotes did not parse in xvfb-run, default values
are appropriate)

added fail-over missing http proxy service (i.e. overload) and quiet
mode

moved snapshot generation out of the html handler to prevent that
existing cache entries cause that the handler is not executed 

more logging

grr

wrap wkhtmltopdf with xvfb if necessary

more logging when failing to create pdf snapshot

added the property timeoutrequests to configuration to disable
TimeoutRequests. The purpose is to test if YaCy runs better on VMs where
there is a limitation of concurrent processes;  see
/proc/user_beancounters in row numproc; this value is limited and should
be low. Try to set timeoutrequests to keep this low. (works only after
restart)

moved network configuration to Use Case submenu; this is necessary
because the definiton of portal peers within the YaCy freeworld network
is otherwise splitted into two different main menus.

replace depreciated Solr DateField.formatExternal with recommended TrieDateField.formatExternal

update to guava.18.0.jar and jsch.0.1.51.jar

skip unused call parameter for hashSentence()

position api icon (ViewFile.html)

update to poi-3.10.1.jar

including small junit test case for WordTokenizer

skip to tokenize punktuation as word in WordTokenizer
remove unused variables in condenser related to Tokenizer

add. use host port parameter in YaCyApp

adjust translation text of  error msg on empty query
(ru: needs correction)

remove obsolete alternate link
fix api link

added image screenshot generator

ignore url errors during search

disabled postprocessing by default. If you read this: please disable
postprocessing in your peer as well: open /IndexSchema_p.html, then
deselect field process_sxt

larger boost fields for ranking

bold words in snippets should not be coloured black in the base style
because there are styles with dark backgrounds which make the bold word
invisible

changed vocabulary navigator object type to TreeMap to get a specific
order into the vocabularies. This is now lexicographic which is not so
much random as a hashed order

added option to change the navbar-default, i.e. usable for dark skins

trying facet.method fc instead of fcs to handle large facets

prevent that a local Solr search and a local RWI search are running
concurrently. When a RWI search result is flushed into the result set,
id does Solr Queries (which replaced the old-style Metadata Queries) and
they are possibly running concurrently to a previously startet Solr
search. Both methods may block each other with IO. To enhance the speed,
they are now serialized. Because the Solr search results may result in
better results using the more advanced and configurable Ranking methods,
this result is preverred over the RWI search result. However, remote RWI
search results are still feeded concurrently into the search result as
well.

fix path lookup to ./defaults/yacy.badwords
(fix of commit https://gitorious.org/yacy/rc1/commit/ee277b9b3e033e8261a97b8334b79059220f113a)

fix empty text facet entry
(noticed on Author facet)

more stacks shall be considered for on-demand loading, not only
deep-depth stacks to prevent "too many open files" problem

reduce number of calls to queue.size() because that may be a bottleneck
during crawling

optimize usage of size() cache

allow for local yacy.stopwords and yacy.badwords list (in DATA/SETTINGS/)
if file in DATA/SETTINGS it is loaded otherwise file in ./defaults is loaded
   (if locale ./defaults/stopwords.xx doesn't exist take solr/lang/stopwords_xx.txt as default)

move yacy.stopwords, yacy.stopwords.de and yacy.badwords.example out of root directory to ./defaults directory

remove redundant toLower for topwords

better delete all files in path when removing host crawl stack

if we have many hosts, use on-demand earlier

prevent division by zero

disabled crazy sleep loop

when importing vocabulary csv files, accept also files without semicolon
and truncate quotes from literals

added hints to ranking to make ranking boosts using vocabularies easier

do not cache search requests to Solr if the result is used for
doublechecking. If a double-check comes from cached results the
doublecheck fails.

use a LinkedHashMap for factes to maintain facet order as given by solr

include domtype to searcheventcache id 
to differenciate between local / global events for reuse of cached events 
fix for http://mantis.tokeek.de/view.php?id=493

added option to enrich vocabularies with synonyms from synonym database

added new solr schema fields which record the occurences of vocabulary
matchings. These matches can be used for result boosting, i.e. if a
document contains words from a specific vocabulary, boost it.

fix for 2-day network stats table: showing 48 instead of 24 hours from
peer history

added option in vocabulary editor to import CSV files with different
encodings (preselected windows-type character encoding which is typical
for CSV files). Fixed also other problems with character encoding in
dictionary files. Automatically generated vocabularies are now also
noted in the API steering.

adjust tag cloud font size calculation 
to limit max font size to ~ TOPWORDS_MAXSIZE 

add a check of java version string >=1.7 to startup class
stopping start with error msg on version < 1.7

added fix to postprocessing: avoid caching of postprocessing collection
to always get fresh lists of documents. This is necessary since the
postprocessing changes the same documents which the
postprocessing-collection query selects.

added high-precision scheduler for API processes. This allows also to
make the execution in dependency of available RAM or CPU load. The
default value for CPU load is 4.0 and the check runs once a minute.

added missing class for latest changes

fix in key enumeration methods for cases where the enumeration is done
in reverse order.

added a input-field for setting 'fileHost'
Set this to avoid error-messages like 'proxy use not allowed / granted'
on accessing your Peer by its hostname.

another fix to ordering of table indexes; fixes also network stats
graphics

remove not used accordion javascript call for facet navs

skip creation of local var in proxyhandler.storetocache

upd NB project.xml to codec-1.9

fix assertation-failure in version-string for Solr-4.10.2 by changing
the assert - hope that is ok
+ add forgotten NB-Projekt-changes

fix for search in case where local peer has no local seed address in
portal mode

added reverse button to tables, by default on now (to see latest entries
first)

added (missing) Tables_p.xml for table xml api

removed unused options from BusyThreads

more enhancements to posprocessing speed

another fix for postprocessing (the query for "" on numeric field did
not work in external solr)

more fixes in postprocessing: partitioning of the complete queue to
enable smaller queries

more concurrency for postprocessing

enhanced postprocessing by usage of a field-list generation to prevent
lazy initialization of the documents. This is useful because the
documents must be read completely anyway.

better scaling of network statistic graphs

replaced old /api/table_p.xml servlet with /Tables_p.xml to avoid double
code

added new index size history image in /Status.html page

added network history in /Network.html?page=5

added debug code for statistics about document attributes related to
domains

RankingSolr: display only available or configured boost fields

replaced input text field with text field for index deletion with query
and replaced GET with POST method. This should make it possible to
tubmit here very large queries for deletion.

bump to httpcore-4.3.3

removed spaces in seedlist.xml to reduce data

added field postprocessing.partialUpdate to settings which can be used
to switch on or off partial updates. Both options should cause the same
result. Default is on.

fix for a ssl bug that appear only in java 7.
The bug was reported in
http://forum.yacy-websuche.de/viewtopic.php?f=23&t=5407&p=30956#p30956
a solution was described in
http://teknosrc.com/javax-net-ssl-sslprotocolexception-handshake-alert-unrecognized_name-solved/
which worked for this example given in the yacy forum

concurrently initialize the error cache; extended also the cache by
factor 10 up to 1000 entries. This error cache is only used to catch up
paused crawls between shutdown+startup

ipv6 fix for api /yacy/seedlist.[json|xml], multiple IPs are now
attached to the seed info. API clients must be adopted. Documentation
will be fixed in
http://www.yacy-websuche.de/wiki/index.php/Dev:APIseedlist

Also added a new retrieval option for seeds, they can now be retrieved
by their name with the get parameter name=<name>

added a timeout on Jetty connectors

do not overwrite yacy.conf in case of an exception
may be a fix for http://mantis.tokeek.de/view.php?id=180

removed warnings

automatically zoom to location/POI

enhanced graphics computation (avoiding long string parsing for colours)

added proper copyright notice to OSM tiles presented at the search
result page

enhanced location search

better profiling of solr queries

added new solr field url_paths_count_i which can be used to enhance the
index browser and maybe also for ranking; possibly also for
SEO-with-YaCy applications.

make browsing of file://z: - paths in index browser easier - this will
now show the root paths on a shared drive

fix-fix for
https://gitorious.org/yacy/rc1/commit/30d4402cd1bbd5629d23562178a049ef7c3b25e9

add filter to citation page and a on/off button
to display only sentences with citations,
while maintaining the sentence number.
Make the filtered list the default in search result citation link

explain crawl denial when not switched to intranet mode

slightly enhanced Network table computation by using a lazy initialized
bitfield for peer flags

refactoring (class name should start with uppercase letter)

added also the NetworkHistory servlet...

added network history graph image /NetworkHistory.png which can show
many different statistics about the history of the peer.

Minor changes:

*) reduced visibility of a method
*) updated comments

fix for values in CrawlProfileEditor table and xml; now the full profile
is available in the xml.

fixed crawl profile xml result which did not show the correct crawl
status.

added another decoration flag to switch off network graphics in crawler
monitor and index browser: decoration.grafics.linkstructure
Please set this to false to remove the graphics from the interface.

added a high cpu cycle monitor to PerformanceQueues

less volume for effect sounds

less load and more ram prerequisite for crawl steps

removed the atmo sound clips because they had been too large

ipv6 fix: avoid that shrinked own ip set is overwritten with (non-valid)
set of local IPs

argh.. adding missing java class for latest audio feature

add link extraction to pdfParser 
this extracts clickable links in pdf and adds it to the list of links

include a test case for this function

this is the corrected comment for commit:
https://gitorious.org/yacy/rc1/commit/aa2e15d846cdee90b70ea882747148b14f257c49

allow url parameter in worktable apicall
 allow url=wwwl?param=a¶m=b (with ?, & encoded)
fix:  http://mantis.tokeek.de/view.php?id=100

fix double adding of  '&' in MultiProtocolURL.escape()

lazy handling of process_sxt field (part of postprocessing)

allow url parameter in worktable apicall
 allow url=wwwl?param=a¶m=b (with ?, & encoded)
fix:  http://mantis.tokeek.de/view.php?id=100

fix double adding of  '&' in MultiProtocolURL.escape()

preserve content_type (mime) if supplied in preference of construct in from file type.
(this eventually can benefit image search by using mime only)

reduce redundant field assignment for Solrdocuments created from URIMetadataNode (URIMetadataNode = SolrDocument with partially assigned fields)

upd to jsoup-1.8.1.jar

open rejected urls in new browser

fix image search expand box, cut-off of 2nd capture line height
tested with IE11 and Firefox 32 (change worked for both to show 2nd line without cutting off height)

+fix charset parameter in metadataImageParser
+update start errMsgTxt to "java 1.7"

typo in javadoc

add html5 autofocus to query input field
(leave onload untouched = redundant, for IE9 http://www.w3schools.com/tags/att_input_autofocus.asp)

adjust Peer-to-Peer/ Privacy switch label 
to display "Peer-to-Peer" as 2nd switch option in active stealth mode

handle  noarchive tag, skip writing page to cache
http://mantis.tokeek.de/view.php?id=44

when pinging other peers, be able to select the right IP option

search result showPicture update search parameter
used parameter &cat=image is obsolete and returns no results
- remove &cat=image and &cat=href references
- remove &tenant= references (unused)
Use contentdom=image and inurl: parameter to make showPicture link display something (open in new window because of used inurl modifier changes original query)

added metadataImageParser for tif and psd (Photoshop) images.
This is a modified genericImageParser adding tif (and psd) support even if java ImageIO plugin for tif is not installed in JDK.
Adds just tif and psd to the available parsers.
Uses the same library to extract metadata, so could eventually be merged with genericImageParser.
All detected metadata are added to the parsed document (potentially some more as with genericImageParser)

use javax ImageIO getReader to add supported image extension/mime
genericImageParser uses javax ImageIO, supported images depend on available plugins for ImageIO package (this is JDK installation specific). Jpeg, png and gif are availabel by default. Tif and others only on avalable plugin (in classpath).
Add supported image type dynamically on startup.

remove unused variable timeout

skip loader wait cycle on concurrent access in nocache configuration.
In nocache config resource is loaded online, leaving no benefit to wait for a faster cache hit.

activated the new apk parser which was already ready but not included in
the parser initialization. To make the apk parser usable, the handling
of application type links had to be modified. Now all documents which
have not a parser attached are placed to the noload-queue while all
other documents are parsed using the associated parser class. This may
have side-Effects on other parsers and the display of different file
classes (images, apps, videos).

added a hack to forward solr search results from an external attached
solr to the YaCy built-in solr search servlet. Its not complete and not
fully correct (there is still a utf8 encoding problem) but it is a way
to get easily requests forwarded through YaCy to an external Solr.

add link to thread pool settings in status panel

fix NPE in ViewFile - show snippet
on document not in index

adjust link to peer in Network list
(www path obsolete)

upd Maven pom (to current dev version)

added warning for not well-formed postprocessing queries

added internal api for partial updates to Solr

added option to reverse-sort YaCy tables (internal API change only)

next development cycle. Please be careful with the usage of next
commits, maybe new and unstable things will come...

catch TimeoutException during ping and do not delete yacy.conf during prereadconfigfile
found a situation after crash (reboot) with existing running semaphore but YaCy not running.
Ping generated exception which finally deleted the conf file (during pre-read procedure)
- change to ping (catch exception solved it)
- additionally removed delete yacy.conf file (if needed we need to make a backup)

better fix for NPE in image search
replace https://gitorious.org/yacy/rc1/commit/8931e14514deff5ab66e7504c45bb78046e5f696